Prototype Filter Design to Minimize Stopband Energy With Constraint on Channel Estimation Performance for OQAM/FBMC Systems

In this paper, we propose a prototype filter design method to minimize the stopband energy of the filter with guaranteed time-domain channel estimation performance for offset quadrature amplitude modulation based filter bank multicarrier (OQAM/FBMC) systems. First, we analytically derive the total mean squared error (MSE) of estimated channel impulse response with two channel estimation schemes, i.e., the linear minimum mean square error and weighted least square, where the impact of intrinsic interference from data symbols is taken into account. Then, we formulate an optimization problem of filter coefficients to minimize the stopband energy with constraint on the total MSE. By exploiting the Heisenberg-Gabor uncertainty principle, the original optimization problem is transformed into an equality-constrained one. Finally, we employ the Lagrange multiplier method to solve the transformed problem and obtain the filter coefficients with the Newton's method. Simulation results demonstrate that the proposed filter outperforms the isotropic orthogonal transform algorithm and extended Gaussian function filters considering both the MSE and stopband energy performances.