Filter Optimization for MFTN-OQAM Systems

In this paper, a novel and effective method for designing the pulse shaping filter for multicarrier faster-than-Nyquist with offset quadrature amplitude modulation (MFTN-OQAM) systems is proposed. The connection between the signal-to-interference ratio (SIR) and the filter coefficients is first established. Then, for a desired overall compression level taking into account compressions in both frequency and time domains, a simple convergence search is suggested to jointly find the optimal values of time and frequency compression factors as well as the filter coefficients to maximize the SIR under a spectrum localization constraint. The obtained results show that higher SIRs, and consequently better bit error rates, can be achieved by the proposed filters over the Martin filter that is commonly used in the filter-bank multicarrier OQAM (FBMC-OQAM) systems (i.e., without time or frequency compression). Moreover, when applying our method to FBMC-OQAM systems, the obtained results show that the original Martin filter is suboptimal as shorter filters are found having the same SIR, which translates to lower implementation cost and latency.


I. INTRODUCTION
The filter-bank multicarrier (FBMC) technique has been studied for many years [1], [2], [3], [4] and is considered as a strong candidate for the physical layer in future wireless communication networks [5], [6]. Compared to the popular orthogonal frequency-division multiplexing (OFDM) technique, a key advantage of FBMC is that it has a much more preferable power spectral density [7], thanks to a well-designed transmit filter [8]. However, FBMC does not provide much data rate enhancement over OFDM. This together with the added complexity are perhaps the reasons that, despite having an excellent spectral property, FBMC has not been widely used in practice.
The multicarrier faster-than-Nyquist (MFTN) technique, first introduced in [9], is a non-orthogonal modulation technique in which one can place data symbols in the time-frequency grid in a flexible manner. Compared to FBMC, MFTN has equally excellent spectrum, but with the added flexibility in controlling the spectral efficiency. This makes MFTN a strong contender against OFDM, FBMC The associate editor coordinating the review of this manuscript and approving it for publication was Jie Tang . and other orthogonality-based multicarrier transmission techniques.
However, the increase in spectral efficiency comes with a price. By reducing the time interval between consecutive data symbols, severe inter-symbol interference (ISI) occurs, which jeopardizes signal reconstruction at the receiver. Similarly, in the frequency domain, by packing adjacent subcarriers closer together, severe inter-channel interference (ICI) is present. Due to these two sources of interference, MFTN could easily suffer from severe performance degradation even when an ideal channel is considered. As such, a well-designed interference cancellation mechanism is needed for MFTN to obtain acceptable error performance. For example, in [9] and [10], the authors have investigated an MFTN system equipped with an iterative receiver that exploits channel coding for interference cancellation. The theoretical analysis suggests that, with a careful choice of system parameters, MFTN can potentially double the throughput over OFDM. A lowercomplexity iterative receiver design is examined in [11] for certain parameters concerning packing data symbols on the time-frequency grid.
It is pointed out that, regardless of the specific interference cancellation technique employed at the receiver, it is always beneficial to design the MFTN system, specifically the transmit filter, in such a way that there is the least amount of interference in the decision variables (corresponding to the transmitted symbols) at the receiver. In other words, it is sensible to design the transmit filter in the MFTN system to minimize the interference in the decision variables before applying any interference cancellation algorithm. This is precisely the main objective of our work.
There are a few works that consider filter design for single-carrier FTN systems, where the main objective is either optimizing the spectrum shape of the FTN signal [12] or maximizing the signal-to-interference ratio (SIR) [13] at the output of the matched filter. Since MFTN systems enable both faster-than-Nyquist signalling and reduced subcarrier spacing, existing filter designs for single-carrier FTN systems do not necessary perform well when applied to MFTN systems.
As for FBMC, various filter designs have been examined and compared in [14]. The squared-root raised cosine (SRRC) filter and the filter based on the extended Gaussian function (EGF) are examples of filters that perfectly satisfy the Nyquist zero-ISI criterion, provided that the filters are of infinite length. With truncation, their finite impulse response (FIR) versions no longer satisfy the Nyquist zero-ISI criterion [15]. In [16], a flexible design of an FIR filter that aims to satisfy the Nyquist criterion as close as possible is provided. The resulting filter, referred to as squared-root Nyquist (m) filter, is shown to outperform the truncated SRRC filter under timing error (manifested by having a cleaner and wider eye diagram). Perhaps the most widely-used filter is developed in [8], and later refined in [17]. It is generally known as a Martin filter and adopted in the PHYDYAS project [18]. More recently, Martin's design was revisited in [19], and further improved to provide a better out-of-band power suppression ability, and consequently a higher SIR. Nevertheless, all the aforementioned filters are not necessarily good choices for MFTN systems. The main reason is that these filters are designed taking into account the Nyquist zero-ISI criterion, which is no longer relevant in MFTN signalling since MFTN deliberately violates the Nyquist zero-ISI criterion to increase the transmission rate. Furthermore, the impact of compression in the frequency domain is also crucial in MFTN and needs to be accounted for.
Motivated by the above discussion, we propose in this paper a simple, yet effective method to optimize the pulse shaping filter for MFTN systems. Adopting offset quadrature amplitude modulation (OQAM) scheme and Martin's filter structure, we form a direct relationship between the SIR at the output of receive filter and the filter coefficients. By obtaining the gradient of the objective function with respect to the filter coefficients, we are able to perform convergence search to find better filters for various settings of system parameters. In particular, our methods finds new filters that not only yield higher SIRs but also have lower implementation cost and latency for MFTN-OQAM systems when compared to the commonly-used Martin filter. As for FBMC-OQAM, which is a special variant of MFTN-OQAM, our optimization pro-cedure obtains a shorter filter whose SIR performance is the same as that of the original Martin filter.
The remainder of the paper is organized as follows. The system model and SIR derivation for MFTN-OQAM are given in Section II. The formulation of the filter design problem is presented in Section III together with the proposed optimization procedure. Section IV presents and discusses the obtained results. Finally, conclusions are given in Section V.

II. SYSTEM MODEL
The baseband-equivalent transmitted signal of the MFTN-OQAM system under consideration is expressed as follows: As in conventional OQAM systems, each complex-valued symbol of duration T is broken up into a pair of real-valued symbols, each of duration T 2 . In the above expression, K is the number of subcarriers, N is the number of real-valued symbols in one block (frame) and g(t) is the transmit pulse shaping filter. The real-valued symbol carried on the kth subcarrier and over the nth time slot is s k,n , whose average energy is normalized to unity, i.e., E{s 2 k,n } = 1. The parameters α and β are the frequency and time compression factors, which control the subcarrier spacing and symbol spacing, respectively.
By sampling with the critical sampling interval T sam = T K , the discrete-time version of (1) is where γ k,n [i] = e j π 2 (k+n) g i − nβ K 2 e j2πkαi K . The expression in (2) suggests a block diagram for discrete-time implementation of the transmitter as shown in Fig. 1, in which realvalued symbols enter the transmitter at intervals of βT 2 . The input symbols are upsampled by a factor of βK 2 to reach the previously mentioned critical sampling interval T sam = T K prior to filtering and modulation. Note that while we treat β as a continuous variable in the design of the system, for practical implementation, it should be selected or rounded such that β K 2 is an integer for implementation. A popular design of the transmit filter g[i] is based on a weighted sum of Q sinusoids, expressed as where L is called the overlapping factor (or filter span in units of symbols) and the length of the filter is KL. In essence, (3) VOLUME 10, 2022 defines the transmit filter through Q frequency coefficients a q .
The scaling factor C is chosen to make an unit-energy filter, i.e., KL−1 i=0 g 2 [i] = 1. It is noted that the filter defined as in (3) is quite comprehensive in the sense that any filter of length KL can be represented as a combination of {a q }. As an example, the simplest filter in the FBMC-OQAM system is designed with Q = 2 and {a 0 , [8]. Since the interference caused by the non-orthogonality among the subcarriers and signalling at a faster-than-Nyquist rate is the main performance limiting factor for the MFTN-OQAM system, we are interested in analyzing the interference power. To this end, we focus on the ideal scenario of having a noiseless channel, which means that the received signal is equal to the transmitted signal, i.e., The block diagram for the front-end of the receiver is also shown in Fig. 1. The output of the matched filter corresponding to the mth carrier and lth symbol is where (m,l) is the cross correlation coefficient between a symbol carried on the kth subcarrier during the nth time slot and another symbol carried on the mth subcarrier during the lth time slot. For convenience, these two symbols shall be referred to as the (k, n)th and (m, l)th symbols, respectively. Regardless of the detection algorithm, the receiver's performance strongly depends on the signal-to-interference ratio (SIR) at the output of the matched filter. Our main objective is to design the transmit filter (and also the receive filter) to maximize the average SIR. Of course, the design should meet specified constraints on the length and spectrum compactness of the filter. The formulations of the average SIR as a function of the filter coefficients and the filter design problem are presented in the next section.

III. PROBLEM FORMULATION AND OPTIMIZATION PROCEDURE
In order to obtain an expression of the average SIR in terms of the filter coefficients, we focus on the cross correlation coefficient, which can be rewritten as Define a vector of the filter's frequency coefficients a = [a 0 , . . . , a Q−1 ] . Then, it is simple to show that (k,n) whose element at the uth row and vth column is given in (8), as shown at the bottom of the next page. Then it can be seen that the ith entry of D (m,l) where G where G 0 = G (m,l) (m,l) is a diagonal matrix that is independent of m and l. Specifically, the uth element on the diagonal of G 0 is This means that the signal power is the same for every symbol in an MFTN-OQAM frame.
On the other hand, for the interference term in (4) We define the average SIR as Unlike FBMC-OQAM, the value of σ 2 m,l varies with l, since σ 2 m,l = σ 2 m,l in general. However, a close examination of (8), (9) and (12) Finally, the optimization of the filter's frequency coefficients to maximize the average SIR can be stated as follows: arg max aξ (a) (15) subject to µ ≥ µ * , where µ ≤ 1 is the localization ratio that controls the spectral leakage and µ * is a specified tolerable level of spectral leakage. This ratio is computed as follows: The localization ratio µ is then computed as It is pointed out that the above optimization problem is invariant to the scaling of a. As such, one can arbitrarily set a 0 = 1 to reduce one unknown variable. The solution obtained by solving the optimization problem can then be normalized to yield an unit-energy impulse response of the transmit filter.
Maximizingξ (a) is equivalent to minimizing f = −ξ (a), which can be expressed as 1 The gradient vector of f is The relevant terms in (18) are derived as follows: and where ∇σ 2 m,l = The final expression for gradient ∇f is given as in (22), shown at the bottom of the next page. With the expression of ∇f , the pseudocode for solving the filter optimization problem is given below.

: return a
It is pointed out that, since f is a non-convex function, the starting point a 0 plays a crucial role in finding the optimal solution. A reasonable strategy would be to begin with the coefficient set of the Martin filter, although one may find a randomized set of coefficients that yields better results when η increases since the optimized set could be far away from the Martin filter's set. Due to the nature of gradient descent search, multiple trials with different starting points are required to assure that the global optimal solution is obtained.

IV. RESULTS AND DISCUSSIONS A. SIR IMPROVEMENTS
In this section, the filters obtained by the proposed optimization are presented for several configurations of the MFTN-OQAM systems. The number of subcarriers is set to K = 32, whereas the overlapping factor is either L = 3 or L = 4, which are the two most widely-used values in FBMC-OQAM. The spectral efficiency is determined as η = R log 2 (M ) αβ (bits/s/Hz), where M and R are the modulation order and the coding rate, respectively. As one can see, for fixed code rate R and modulation order M , different gains in spectral efficiency can be obtained for MFTN by adjusting the product αβ. In contrast, the spectral efficiency of FBMC is fixed at R log 2 (M ) (bits/s/Hz). Furthermore, since M and R do not affect the calculation of the average SIR, one can set R = 0.5 and M = 4, which means s k,n ∈ {−1, 1}. For all the results presented in this section, the tolerable level of spectral leakage is specified as µ * = 95%. First, the optimization algorithm is performed for the case that the number of filter's frequency coefficients is equal to the overlapping factor, i.e., Q = L. Although not necessarily an ideal choice, it has been widely adopted in the literature of FBMC-OQAM [8]. In particular, for the case Q = L = 3, the frequency coefficients of the Martin filter obtained in [8] for FBMC-OQAM is a 0 = [1, −0.91143783, 0.41143783].
Practically, spectral efficiency (which is controlled through the product αβ) is often specified as one of the design targets, and then one should choose α and β accordingly to maximize the SIR for the specified spectral efficiency. Therefore, with each value of η, the optimization algorithm refines a n and outputs the optimized a as shown in Table 1. The table also compares the SIR values between the optimized filter and the original Martin filter for the same set of compression factors (α, β). It can be seen that for all combinations of (α, β), our optimized filters always perform better than the Martin filter, especially when η is close to 1. For example, a gain of 1.5 dB is achieved when using the proposed filter for η = 1 (i.e., a FBMC-OQAM system). On the other hand, a gain of 2.1 dB is obtained by the proposed filter for η = 1.1.
Similar SIR improvements are obtained with the filters optimized with Q = L = 4 and reported in Table 2, in which the largest SIR gain is observed when η = 1.1. The coefficients of the original Martin filter designed for FBMC-OQAM systems are a 0 = [1, −0.97195983, √ 2/2, −0.23514695] [8]. Since the filter length is larger in this case (with L = 4), the average SIRs are higher than those in Table 1 It is worth pointing out that, even for an FBMC-OQAM system, the proposed filter always provides SIR improvement over the original Martin filter, regardless of filter length.
As discussed before, the number of filter's frequency coefficients Q needs not be the same as the overlapping factor L, nor does it affect the length (hence complexity) of the filter in the time domain. In fact, one should use a large value for Q in order to have more flexibility in optimizing the spectrum of the filter, and consequently achieve higher SIR. This is easily explored with the proposed optimization algorithm and the results are presented in Fig. 2 for η = 1 (corresponding to an FBMC system) and two different values of L. It can be seen that, for the case L = 3, using Q = L = 3 delivers the maximum SIR of 43 dB, whereas the maximum SIR increases to 68 dB with Q = 7, i.e., a gain of 25 dB over the conventional Although the optimization algorithm can also find optimal filters for the cases η > 1 (corresponding to MFTN systems), no significant improvements in SIR are observed by increasing the number of filter's frequency coefficients (Q > L) in those cases. As a result, our proposed method can be used to obtain optimized  filters for MFTN-OQAM by setting Q = L, since using higher values of Q is practically unnecessary.

B. FILTER RESPONSES
In this part, the original Martin filter and the optimized filter are compared in terms of ISI immunity and spectrum localization. Consider an FBMC-OQAM system (i.e., an MFTN-OQAM system with α = β = 1) with K = 32 and L = 3. For each design, the impulse response of the cascaded transmit-receive filter is plotted in Fig. 3. Recall that the original Martin filter achievesξ = 43.4 dB, whereas the optimized filter hasξ = 68 dB. As expected, both cascaded filters peak at KL + 1 = 97. At samples KL + 1 + iK , the optimized filter satisfies the zero-ISI criterion better than the original filter, with amplitude difference of 10 −3 .   Compared to the original Martin filter, the optimized filter has poorer spectral localization as shown in Fig. 4. Specifically the stopband of the optimized filter is wider and there is a notable sidelobe next to the passband of the filter. However, it is stressed that the relevant and important assessment is the power spectrum density (PSD) of the FBMC-OQAM signal, which is shown in Fig. 5 for both filter designs. Although having higher spectral sidelobes, the overall PSD obtained with the proposed filter has very sharp transition band and very low amount of out-of-band power emission.
Similar observations can be made from Figs. 6, 7 and 8 for the case L = 4. In Fig. 6, the cascaded response of the optimized filter is infinitesimal at samples KL + 1 + iK , with the difference of 2 × 10 −4 . Figs. 7 and 8 show some deteriorations of the magnitude response and PSD of the optimized filter compared to the original Martin filter. However, such deteriorations can be well justified by the much better ISI immunity, and consequently by the much better overall SIR.

C. BIT ERROR RATES
Next, we investigate how the SIR improvement translates to the bit error rate (BER) performance when the optimized   filters are compared to the original Martin filter. To this end, we obtain the BER over an additive white Gaussian noise (AWGN) channel for systems that implement a rate R = 1/2 convolutional code with constraint length 10, whose generator matrix is [2473,3217] in octal form. The coded sequence is mapped to symbols s k,n using the standard QPSK constellation and Gray mapping. At the receiver, the soft-input hard-output Viterbi algorithm is implemented to decode the log-likelihood ratio of the demodulated binary sequence. It is noted that no interference cancelation method is implemented before channel decoding. This is to observe the gross performance improvement in terms of BER with the optimized filters.
As can be seen in Fig. 9, when η = 1 and L = 4, the use of both original and optimized filters leads to virtually identical BER performance. This is because the average SIR resulting from using these two filters are very high, making the interference insignificant compared to thermal noise. For the same reason, the BER performance of the system using the optimized filter with L = 3 (which achieves an SIR of about 68 dB) is equivalent to the two filters having L = 4. This comparison means that our filter optimization algorithm reduces the filter length by 25%, from 4K to 3K , without sacrificing the error performance.
When η = 1.1, the SIR gain helps to improve the BER. Specifically, it can be seen from Fig. 9 that the optimized filters with both L = 3 and L = 4 deliver better BER performance compared to the original filter designed with L = 4. At the BER level of 10 −5 , a 4 dB gain is achieved with the optimized filter designed with L = 4 over the original filter of the same length. Note that the SIR improvement between these two filters is about 2.2 dB as discussed before. Similar observation can be made when η = 1.2. The SIR improvement of 1.1 dB translates to 3.2 dB gain in E b /N 0 at the BER level of 2 × 10 −5 . The results clearly show that the optimized filters are preferred over the original filters when the system is designed for a higher spectral efficiency (η > 1).

V. CONCLUSION
This paper has formulated and solved the filter design problem for multicarrier faster-than-Nyquist (MFTN) systems with offset quadrature amplitude modulation (OQAM). With the objective of maximizing the signal-to-interference ratio (SIR) at the output of the matched filter, the frequency coefficients of the transmit filter are found that meet a criterion on spectral localization and for a fixed spectral efficiency. The obtained results showed that the optimized filters can boost SIR up to 25 dB in the orthogonal multicarrier systems (FBMC-OQAM), and up to 2.2 dB for non-orthogonal MFTN-OQAM systems that implement compression factors to increase the spectral efficiency. The BER results have also confirmed the usefulness of the optimized filters and showed that, for the non-orthogonal MFTN-OQAM systems, the SIR improvement translates to a larger power gain by using the optimized filters. Future works could investigate the MFTN-OQAM systems with the optimized filters under other channel models, as well as with more advanced receivers. The extension of this work to multiple-input multiple-output (MIMO) systems is interesting, considering that filter design can help reduce intrinsic interference that is more problematic with MIMO channels. Such an extension is beyond the scope of this work and deserves a further and extensive study.