Genetic Algorithm Based Prototype Filter Design for Oriented Side Lobe Energy Suppression in FBMC System

The prototype filter design problem is investigated for the filter bank multicarrier (FBMC) system of the fifth generation (5G) physical-layer wireless communications. In order to further suppress the side lobe energy within a certain frequency range, different constraint factors need to be introduced to meet the various side lobe energy suppression demands. In this paper, we formulate a dual-objective optimization problem which minimizes the stopband energy with constrained factors and subjects to the ISI/ICI constraints. Considering the uncertain constrained factors, a suboptimization problem is proposed by constraining the total stopband energy and the side lobe energy of the first segment to minimize the side lobe energy of the second or the third segments. Then, the nested sequential quadratic program-genetic algorithm (NSGA), one of the artificial intelligence (AI) aided algorithms, is introduced to obtain the optimal solution of the dual-objective problem, in which the genetic algorithm (GA) is applied to acquire the optimal constrained factors and the sequential quadratic program (SQP) is applied to acquire the optimal filter coefficients. Numerical results validate that the proposed method can achieve orientated side lobe energy suppression at specified segments for satisfying different side lobe energy suppression requirements with the confirmed algorithm convergence.