Filter optimization of out-of-band radiation with performance constraints for GFDM systems

Generalized frequency division multiplexing (GFDM) is a promising modulation scheme featuring low out-of-band (OOB) radiation, which is achieved through the use of prototype filters. However, using prototype filters can make GFDM systems non-orthogonal, incurring in-band performance degradation in receiver mean square error (MSE) and symbol error rate (SER) compared to that achieved through orthogonal frequency division multiplexing (OFDM). In this paper, a filter optimization algorithm that minimizes OOB radiation while maintaining good in-band performance is proposed for GFDM. Through a matrix-based characterization of GFDM systems, the filter design problem is formulated as a nonconvex problem. After some transformations, an algorithm in which two convex problems are solved iteratively is proposed to tackle the original problem. Simulation results show that under the same spectral efficiency, optimized filters perform the best in terms of both OOB radiation and SER performance, compared to OFDM and prototype filters existing in the literature.