Prototype Filter for QAM-FBMC Systems Based on Discrete Prolate Spheroidal Sequences (DPSS)

In conventional wireless communications, cyclic-prefix orthogonal frequency division multiplexing (CP-OFDM) has been adopted as the baseline multicarrier scheme. Despite their corroborated merits, they are explored only in the asynchronous and strictly orthogonal scenario. To overcome the limitations observed in CP-OFDM, to support the constraints imposed by different 5G scenarios, and also to improve robustness against channel impairments, several waveforms have been investigated. Quadrature amplitude modulation associated with filter-bank multicarrier (QAM-FBMC) has been an auspicious technology for 5G communication systems and beyond. The main feature of QAM-FBMC is its capacity for high spectral confinement, which is possible thanks to the per-subcarrier filtering. Aiming to improve the QAM-FBMC performance, in this paper, we propose a prototype filter design based on the discrete prolate spheroidal sequences (DPSS), also known as Slepian sequences. The presented filters were obtained by optimizing the weights that are used to compose the desired prototype filter to such an extent that they minimize the intrinsic interference of the system. At the same time, these weights keep spectral confinement of the filter for a previously determined bandwidth limitation. Simulation results show that the optimized filters achieve higher intrinsic interference attenuation than the competitors. Indeed, we can confirm the improvement in the system performance brought by the usage of the optimized filters through the bit error rate (BER) evaluation. Furthermore, the proposed method is flexible thanks to the suitability of its parameters.


I. INTRODUCTION
Waveform design for post orthogonal-frequency-divisionmultiplexing (OFDM) systems, e.g., the fifth generation (5G) and beyond (B5G), is a challenge due to the amount and the nature of the services that must be provided [1]- [3]. Among the recent candidates we can cite the offset-quadrature amplitude modulation filter-bank multicarrier (OQAM-FBMC) [4], [5]. Its major characteristics lie in keeping only the real part of the orthogonality condition, crucial for OFDM systems, and in using a persubcarrier pulse-shaping to reduce out-of-band emissions (OOBE). Consequently, it provides flexibility in waveform design when compared to OFDM and allows heterogeneous The associate editor coordinating the review of this manuscript and approving it for publication was Haipeng Yao . services, which is also a requirement of 5G/B5G communication systems. Unfortunately, the one-dimensional intrinsic interference observed in OQAM-FBMC systems becomes a source of problems when combining known techniques, like multiple-input multiple-output (MIMO) and maximum likelihood (ML) detection [6], [7].
Pulse-shaping OFDM systems with good time-frequency localization (TFL) have been proposed in [8], [9] to cope with doubly-selective channels. In order to keep the perfect reconstruction of the data, the authors in [10], [11] have considered a time-frequency lattice associated with an OFDM system with a lattice density (1/TF) lower than 1 (i.e. TF > 1), where T is the symbol period and F is the subcarrier frequency spacing. To overcome the drawbacks observed in OQAM-FBMC, a QAM-based FBMC scheme (QAM-FBMC) with a single prototype filter has been proposed in [12]. It can be seen as a generalized OFDM with a time-frequency lattice of density equal to one. A more complex structure of QAM-FBMC, composed of two prototype filters, has been proposed and studied in [13] and [14]. An even more complex structure, based on several prototype filters, was proposed in [15]. The use of more than one prototype filter reduces the interference but requires more elaborated receivers in order to mitigate the observed residual intrinsic interference.
Several researchers have been working on different techniques for enhancing QAM-FBMC performance. A basic iterative interference estimation and cancellation have been proposed in [16]. The authors in [17] proposed an interference mitigation scheme based on precoding and decision feedback equalization. In [18] researchers proposed an iterative interference cancelation (IIC) receiver to minimize the interference. Also, the authors of [19] proposed a nonlinear receiver, that uses IIC and parallel interference cancellation (PIC) concepts in multiple-input multiple-output (MIMO) systems with minimum mean square error (MMSE) receiver.
We can also find in the literature several works on prototype filter design for QAM-FBMC systems in an attempt to enhance its performance. In [13] the authors proposed the prototype filter called Type I, which is based on the minimization of the self inter-symbol interference (ISI) subjected to the fall-off-rate limitation. As the obtained filter in [13] was poorly concentrated in the time-domain, another waveform design considering time domain localization based on a single prototype filter was suggested in [20] at a cost of a slight increase in interference. In [20] the real coefficients were obtained from the minimization of self signal-to-interference ratio (self-SIR), under the constraints of fall-off rate and time dispersion. By using a global optimization algorithm, the authors proposed three different filters. The number of frequency-domain coefficients of their filters was set as 7, 11, and 15, which led to the filters named Case A, Case B, and Case C, respectively. We can also cite filter design proposals based on carrier frequency offset (CFO) [21]; filter design proposals specifically for minimum mean square error (MMSE) based receiver for a given signal-to-noise rate (SNR) value [22]; and, more recently, a design based on the ISI minimization subject to OOBE limitation [23]. However, the performance evaluations of these prototype filters remain unsatisfactory, since each proposal considers different parameters for the optimization, such as specific SNR, bandwidth, or class of the receiver. Consequently, a direct comparison of their performances is questionable.
The authors of [13] have proposed the use of two prototype filters to reduce interference. However, the bandwidth of the second filter is significantly larger than that of the first filter. This change makes the comparison to the case of using one prototype filter unfair. Besides, the complexity of the system also increases.
We can look at prolate spheroidal wave functions (PSWFs) proposed by Slepian [24], [25], as a possible solution for QAM-FBMC prototype filter design. David Slepian proposed a solution for the time and frequency localization through a set of functions denominated prolate spheroidal wave functions (PSWFs), which are maximally energy concentrated in a given time and bandwidth limitation. It is known that the Slepian solution for the discrete-time problem, called discrete prolate spheroidal sequences (DPSS), is the equivalent version of PSWFs in discrete-time domain [26], consequently they have the same properties. Both DPSS and PSWFs have been widely studied [27]- [31].
We propose, in this paper, a prototype filter specifically designed for QAM-FBMC systems considering a matched filter at the receiver. We consider the discrete prolate spheroidal sequences (DPSS) in order to achieve good energy concentration for a given time and bandwidth limitation. However, differently from traditional filter designs, we did not minimize the OOBE, but the intrinsic interference of the system. We compare the performance of the obtained filters to some known filters from the state of the art.
The proposed prototype filters show high interference attenuation and, as expected, significantly improve the system BER performance. In addition, since the bandwidth of the desired filter is adjustable through the DPSS parameters, the proposed procedure shows flexibility in terms of spectrum confinement.
The contributions presented in this paper are: • A new prototype filter for FBMC-QAM based on the most energy concentrated DPSS sequences.
• A new technique to optimize prototype filters for QAM-FBMC based on DPSS that minimize the intrinsic interference for a given bandwidth limitation.
• A system performance comparison of the proposed prototype filters with the ones in the literature considering both AWGN and fading channels.
For the sake of clarity, we start by describing the QAM-FBMC system and deriving its intrinsic interference in Section II. Then, we give an overview of PSWF and DPSS in Section III. We present our DPSS-based prototype filter design as an optimization problem in Section IV. The optimization results are presented in Section V and the performance evaluation of the proposed filters in Section VI. The conclusions are drawn in Section VII.

II. SYSTEM MODEL OF QAM-FBMC
Let us define a k ,n as the transmitted symbol allocated at the k -th subcarrier and transmitted in the n -th QAM-FBMC symbol. Thus, the QAM-FBMC transmitted signal, also known as the Weyl-Heisenberg system of functions [39], VOLUME 10, 2022 can be given by where M is the number of subchannels, g(t) is the unit-energy prototype filter with overlapping factor K (i.e. length L g = KM ), T is the symbol time, and F is the subcarrier frequency spacing.
Considering additive white Gaussian noise channel (AWGN), the continuous-time received signal can be described as where v(t) is Gaussian white noise. At the receiver, the output of the n-th symbol of the k-th filter can be expressed as Let us recall the ambiguity function definition [40] where * denotes complex conjugate. By setting n = (n−n ) and k = (k − k ) we can derive the function A g (T n , F k ) as Thus, by using (5) we can rewrite (3) as As an example, we show in Fig. 1 the ambiguity function of the prototype filter case C proposed in [20] for QAM-FBMC systems with a single filter. The red marks represent the rectangular lattice structure of the QAM-FBMC system. These points, except for the one in the center (( k , n ) = (0, 0)), indicate where we should sample the ambiguity function in order to obtain the intrinsic interference coefficients which will be used to compute the system interference.
Let us define the intrinsic interference coefficients as represented by the red points in Fig. 1. Therefore, the total interference over the k-th subcarrier at the n-th time instant can be calculated using the interference coefficients to weight the neighbor symbols as follows As we can see, these coefficients are essential to calculate the intrinsic interference, since the quantity of k , n , when (k , n ) = (k, n), is dedicated to the interference that comes from the surrounding symbols (k , n ).

III. DISCRETE PROLATE SPHEROIDAL SEQUENCES (DPSS)
The problem of confining a signal in time and frequency domains simultaneously has been discussed in digital signal communication systems for a long time [41]. The energy time-concentration level of a signal can be measured as the fraction of the signal's energy lying in a time interval of length 2T 0 . This measure can be mathematically written as If g(t) is strictly time-limited to the interval [−T 0 , T 0 ] the energy concentration level P will achieve its maximum value of one. As time and frequency are inversely related, the Fourier transform of a time-limited function is equal to an infinitely wide bandwidth.
The time and frequency concentration issue, which consists in maximizing P while keeping g(t) band-limited, was first studied by Slepian, Pollak, and Lindau [24], [25]. They developed a set of band limited functions ψ p (t, b) that are maximally concentrated in a given time interval [24]. These functions are called Prolate Spheroidal Wave Functions (PSWFs), also known as Slepian functions.
The approach used by Slepian is based on angular and radial solutions in spheroidal coordinates of the first kind to the Helmholtz wave equation S 0 p (b, t) and R 0 p (b, 1), respectively. Hence, the PSWFs can be expressed as [24], [42] where λ p (b) is given by The PSWFs are dependent on the continuous time parameter t, on the order p of the function, on the time parameter T 0 , and also on the parameter b which is defined as where B is the bandwidth of ψ p (t, b) of a given order p.
, can be defined as the normalized eigenfunctions of the following integral equation [26], [27]: The sinc function in (12) can be regarded as a symmetric Toeplitz operator kernel, and the integral of ψ p (t, b) multiplied by this kernel can be called a symmetric Toeplitz operator [26]. Also, λ p (b) are the eigenvalues of the sinc function kernel, which can also be seen as the index of energy concentration on the interval [−T 0 , T 0 ] [43]. If λ 0 (b) denotes the largest eigenvalue of (12), then its corresponding eigenfunction, ψ 0 (t, b), is commonly called Slepian window, or prolate spheroidal window in the continuous-time domain.
In addition, we can also examine the time/frequency concentration problem in discrete time. For a given time interval 2T 0 with L samples, and for each p = 0, 1, 2, . . . , L − 1, Slepian defined the discrete Prolate Spheroidal Sequences (DPSS) φ for c ∈ Z, and λ p (L, B) are the eigenvalues of the sinc function kernel [44]. This is the discrete time representation of the system described in (12). In our application, we are interested in the sequences φ  is the unique sequence that is time-limited and the most frequency-concentrated, φ (1) (L, B) is the second sequence having maximum energy concentration among the DPSS, and orthogonal to φ (0) (L, B), and so on. Therefore, we can rewrite (13) in a matrix approach as These sequences are thus seen to be the eigenvectors of the L × L matrix Q(L, B) with elements [44], [45]: In Fig. 2 we illustrate some of these sequences. PSWFs should not be confused with the solution of the discrete-time energy concentration problem, the DPSS. DPSS are much simpler to calculate, and they are known only in a finite interval [27].

IV. DPSS-BASED PROTOTYPE FILTER DESIGN
The features presented here suggest the potential utility of DPSS for filter-bank multicarrier systems. The first prolate sequence, φ (0) (L, B), is also known as the classical prolate window. It provides the optimum frequency-concentrated pulse for a given filter length and bandwidth [44]. It is defined as the eigenvector corresponding to the largest eigenvalue of the matrix Q (L, B), usually called Slepian window, or prolate spheroidal window in the discrete-time domain.
Another possible solution that we can think of for filter design is the Kaiser window [46], [47], which employs Bessel functions in order to obtain an approximation of the prolate window. Although it has the advantage of its formulation through a closed-form expression, it is sub-optimal in terms of out-of-band leakage when compared to the prolate window.
Knowing favorable characteristics of discrete prolate spheroidal sequences, we propose a prototype filter design technique based on DPSS, and we optimize the obtained filter. For this, we consider the spectral confinement, through the band-time parameter of DPSS, and minimize the intrinsic interference of the QAM-FBMC system by using its ambiguity function. We start by designing a function that represents the desired prototype filter using the DPSS.
As the eigenvalues can be regarded as the index of energy concentration of the DPSS [24], our proposal consists in selecting the eigenvectors φ (p) (L g , B) of matrix Q(L g , B) whose eigenvalues λ p (L g , B) are higher than a certain threshold γ . The first eigenvalues have strong energy-concentration behavior, they are very close to 1, and the remaining eigenvalues are close to 0 [24]. By doing this process, we select the N e most energy concentrated sequences φ (p) (L g , B) for a given band limitation B.
Finally, the proposed prototype filter can be described as a weighted sum of the N e sequences as follows: where g = g[0], g [1], . . . , g[L g − 1] is the discrete response of g(t) with g[m] = g m T M , and w p is the weight coefficient.
From the definition of the desired prototype filter given in (16), and taking into account the interference of the system described in (7), we propose an optimization problem to find the best prototype filter for QAM-FBMC systems. In this sense, we want to find the optimal weights w p of the prototype filter that minimize the energy of the intrinsic interference coefficients k n subject to the filter energy constraint. So that, we formulate our optimization problem as where the cost function is the discrete version of the ambiguity function given in (5). In order to solve the above optimization problem, we use in this work the interior point (IP) method [48]. We have done the optimization several times, each time with different random initial weights w, and we select the best-obtained result, i.e. the one with the smallest amount of interference. Our obtained prototype filters are denoted as DPSS-based filters (DPSSb).
Algorithm (1) describes the procedure adopted to solve the optimization problem (17).

V. OPTIMIZATION RESULTS
In this section, we present the obtained optimization results for a specific FBMC-QAM system settings. We set the overlapping factor of the desired filter as K = 4, and the number of subcarriers M = 128, which results in a filter length L g = 512. compute Q(L g , B) as in (15) 4: select the eigenvalues greater than γ 5: select the corresponding eigenvectors φ (p) (L g , B) 6: for i = 1 to i max do 7: w ∼ CN (0 N e ×1 , I N e ) 8: w ← w/ w 9: compute g according to (16) 10: w ← solution of problem (17) through IP 11: compute g according to (16) 12: interf = (k ,n ) =(k,n) | k n | 2 13: if interf < interf min then 14: interf min ← interf 15: g opt ← g 16: end if 17: end for 18: return g opt 19: end procedure For a good compromise between N e and OOBE, we used in our designs the threshold γ = 0.99 for the eigenvalues. For a fair comparison to the state-of-the-art filters known in the literature, we have determined the bandwidth as B = 7/M for the optimization of the filter that was compared to the Case C filter proposed in [20]. For the comparison to the Type I filter proposed in [13] we set the bandwidth B = 1.7/M . The compared filters have approximately the same bandwidth. These filters have been chosen due to their characteristics, they were not optimized for a specific SNR or CFO. To the best of our knowledge, there is no prototype filter in the literature for QAM-FBMC with bandwidth B = 3/M , but we have also optimized a prototype filter with this bandwidth in order to demonstrate the proposed procedure flexibility.
In the case of B = 1.7/M , by setting γ = 0.99, we have found N e = 3 eigenvectors, therefore we had to optimize 3 weights w p . In Fig. 3 we present the ambiguity function of the DPSS-based optimized filter. As we can notice, the interference spreads differently in time and frequency domains. It spreads more in time than in the frequency domain.
We have also compared the frequency response of the optimized DPSSb B = 1.7/M filter with the frequency response of Type I filter [13] in Fig. 4, both of them normalized to unit energy. As expected, due to the choice of B, they have approximately the same bandwidth. However, their energy distribution is not the same. It happens due to the filter design procedure, which is focused on the intrinsic interference minimization, not in OOBE, as observed in classical filter designs.
The optimization procedure was also applied for the comparison with the Case C filter [20]. We present in Fig. 5 the ambiguity function of the optimized DPSSb B = 7/M filter.  For this optimization, considering γ = 0.99, the number of selected eigenvectors was N e = 25.
The ambiguity function of the DPSS-based 7/M optimized filter is not symmetric. This fact comes from the complex values of the optimized weights. As expected, comparing this ambiguity function to that for B = 1.7/M , we can notice an increase in the interference spreading in the frequency domain. It comes from the fact that, in this case, we are optimizing a filter with bandwidth far greater than the subcarrier spacing 1/M .
The comparison between the frequency response of DPSSb 7/M filter and Case C [20] filter is also presented in Fig. 6. Once again, the optimized filter and the known filter have approximately the same bandwidth, however, their energy distribution is not the same.
Despite achieving higher attenuation at the edges of the transmission band, the passband of the DPSSb 7/M is not  flat. This result comes from the fact that the bandwidth of DPSSb 7/M comprises more than one subcarrier spacing (1/M ). As we minimize the interference coefficients k n , which are spaced by one subcarrier spacing, some of them are inside the considered passband of the optimized filter.
In order to make a comparison of the prototype filters, we present, in Table 1, some characteristics of the known and the proposed filters. We compare the inter-symbol interference (ISI) observed in a QAM-FBMC system when using all of these prototype filters. Besides, we evaluate the outof-band energy (OOBE) of them considering their bandwidth and we also classify them by the nature of their coefficients.
By increasing the desired bandwidth it is possible to decrease the ISI of the optimized prototype filter. Despite having approximately the same bandwidth, the DPSSb 1.7/M filter presents smaller ISI than the Type I filter, however, it happens at a cost of higher OOBE. Unlike what happens   with the DPSSb 1.7/M filter, the optimized DPSSb 7/M has approximately the same bandwidth as that of the Case C filter. Its ISI is smaller and also its OOBE. This fact can be explained by the complex nature of the optimized filter coefficients, whereas Case C filter coefficients are real. It is important to point out that the proposed DPSSb 7/M filter achieves the highest interference attenuation among the optimized filters, and the smallest OOBE.

VI. PERFORMANCE EVALUATION
In this section we evaluate the performance of the QAM-FBMC system when using the optimized DPSS-based (DPSSb) prototype filters and compare the results with those obtained when using the prototype filter Case C proposed in [20] and Type I proposed in [13].
The performance of the prototype filters was evaluated through the bit error rate (BER) over Additive White Gaussian Noise (AWGN) channel and also over the pedestrian channel, defined by 3GPP [49]. The presented results are obtained by averaging over different channel instantaneous realizations. The performance evaluation was performed over different modulation levels: 4-QAM, 16-QAM and, 64-QAM.
In order to enhance the overall system performance, the iterative interference cancellation (IIC) [16], [18] was applied  at the receiver. For the sake of clarity, we also present the Genie Aided performance where the interference is assumed to be perfectly known at the receiver.
In Figs. 7 and 8, we present the BER performance of QAM-FBMC system over AWGN channel with 4-QAM and 16-QAM modulations respectively. We compare the results using DPSSb 1.7/M optimized filter with the results of Type I filter. In the case of 4-QAM modulation, the system performance can be significantly improved by the use of the optimized filter. However, with just one IIC iteration it is possible to achieve the same performance for both filters, which is the same as the Genie Aided. Even observing a significant degradation in the performance for 16-QAM modulation compared to the 4-QAM modulation, the proposed filter outperforms the Type I filter. In this case, applying the  IIC technique, even with two IIC iterations, the proposed DPSSb 1.7/M outperforms the Type I filter.
The QAM-FBMC system was also evaluated over pedestrian channel. Figs. 9, 10, and 11 compare the performance of DPSSb 1.7/M and Type I filters when using 4, 16, and 64-QAM modulation respectively. In the case of 4-QAM modulation, even without using IIC technique, the DPSSb 1.7/M performance is very close to that of Genie Aided, whereas for Type I one IIC iteration is necessary to achieve the same performance.
Considering 16-QAM and 64-QAM modulation, we can observe an overall system degradation as expected. Once again the optimized filter outperforms the Type I filter and achieve the Genie Aided performance with 16-QAM modulation by performing 3 IIC iterations. Although for 64-QAM modulation the system suffers from high degradation, we can still confirm the superiority of the DPSSb 1.7/M filter, since VOLUME 10, 2022  it achieves almost the same performance of Type I after 5 IIC iterations but with just 3 iterations.
As we did not find in the literature a prototype filter with approximately the same bandwidth to make a fair comparison to the DPSSb 3/M , we present its performance in Figs. 12, 13, and 14 compared to the Genie Aided performance in the cases of 4-QAM, 16-QAM, and 64-QAM modulation respectively.
As we can see, for 4-QAM modulation we achieve the same performance as the Genie Aided, even without IIC. The Genie-Aided performance is achieved after 3 IIC iterations when the 16-QAM modulation is applied. In the case of 64-QAM modulation, the improvement obtained with our proposed DPSSb filter applied in tandem with IIC is significant.
For the comparison to the Case C filter, we have optimized a prototype filter DPSSb with bandwidth B = 7/M . We present, in Figs. 15, 16, and 17, the system performance when applying 4-QAM, 16-QAM, and 64-QAM modulations, respectively.
In the case of 4-QAM modulation, the Genie-Aided performance is achieved even without IIC. Besides, both filters have the same performance until approximately 34dB of E b /N 0 . After that, the performances are slightly different. One IIC iteration is needed for achieving the same performance of Genie Aided in the case of 16-QAM modulation for both filters. However, without IIC the optimized filter outperforms the Case C filter. Once again we can notice a slight difference between the performances for E b /N 0 greater than 34dB. In the case of 64-QAM modulation, the overall system performance deteriorates. Despite that, DPSSb 7/M filter achieves better results when compared with the Case C filter, until three IIC iterations, when they have very similar performance.

VII. CONCLUSION
In this paper, we propose a novel method for the design of prototype filters for QAM-FBMC systems which are based on DPSS. We consider the spectral confinement through the band-time parameter of DPSS and minimize the intrinsic interference coefficients of the QAM-FBMC system. As a result, the obtained filters achieve smaller ISI than the competitors found in the literature. Furthermore, our proposed design shows to be flexible considering their adjustable parameters as bandwidth, overlapping factor, and filter length.
The performance of the proposed DSSSb filters was evaluated through the BER of the system and compared to known filters, such as Case C and Type I, which are strong competitors among the state of the art. We consider different modulation levels, AWGN, and pedestrian channels, and for all the considered scenarios the DPSSb-optimized filters outperform the competition, with and without interference cancellation.