A PPN-Based Improved QAM-FBMC System With Jointly Optimized Mismatched Prototype Filters

Recently, it is shown that a spectrally-efficient low-complexity quadrature amplitude modulation (QAM) filter-bank multicarrier (FBMC) system can be designed by relaxing the constraints on the time-frequency (TF) product at the transmitter (TX) and the time-domain localization at the receiver (RX). However, it turns out that the attainable signal-to-interference-plus-noise ratio (SINR) is not satisfactory enough to support certain high-order modulations. This is because, given the PHYDYAS TX prototype filter, only the RX filter is optimized under a sparsity constraint. In this paper, we propose an improved QAM-FBMC system to further reduce self-interference by jointly optimizing the TX and RX prototype filters. The sparsity constraint on the RX prototype filter is now removed, and a new polyphase network (PPN)-based structure is introduced to maintain the complexity at the RX to almost the same level. The joint optimization is formulated as an approximate SINR maximization and converted to a line search, under the constraint on the fall-off rate of the TX prototype filter for high spectral confinement. For each search-parameter value which is a lower bound on the post-processing signal-to-noise ratio (SNR), the prototype filters are optimized to maximize the signal-to-interference ratio (SIR). The line search stops at a saturation point of the SINR, and the pair of fixed TX and RX filters obtained at the point is used for all SNR ranges. Numerical results show that the prototype filters combined with the PPN-based structure achieve low self-interference and lead to a spectrally-efficient low-complexity QAM-FBMC system.