Deep-Learning for Generic Blind-Joint Channel Equalization and Power Amplifier Post-Distortion

One of the major problems faced in digital communication systems is Inter-Symbol Interference (ISI), induced by the propagation channel in the single carrier based systems. Classic digital equalization techniques based on pilot training sequences become tedious in the presence of nonlinear power amplifiers. The existing techniques for digital pre-distortion need a high transmitter computation complexity and those for the post-distortion require pilot overhead. In this review, we focus on fully generic blind processing for both channel equalization and power amplifier post-distortion. We propose a receiver based on two complex-valued neural networks (CV-NN). The first CV-NN is dedicated to generic blind equalization (GBE) to mitigate ISI. The second one is used for generic blind post-distortion compensation (GBPDC) of the power amplifier nonlinearity (PANL). The GBE and the GBPDC have no prior information about the transmission channel, the used constellation, and the PA model. For the first CV-NN, we consider an updated probability density fitting (PDF) based criteria, corresponding to many assumed possible constellations, that are used jointly with an automatic modulation classification (AMC) based on the k-nearest neighbors (KNN) algorithm. For the second CV-NN, we use the final updated PDF criterion resulting from the first CV-NN training process. Numerical results show that our generic blind Deep learning-based signal receiver formed with the two CV-NNs is effective in alleviating the two coupled signal distortions: the ISI and the PANL. Compared with the state-of-the-art methods, banking on a supervised post-distortion compensation and channel equalization, the proposed generic blind DL-based scheme exhibits good detection performance.


I. INTRODUCTION
The nonlinearity of power amplifiers (PA) remains one of the serious constraints for the fifth generation (5G) and beyond cellular systems since signals have extremely high peak-toaverage power ratio (PAPR).Moreover, channel estimation and equalization remain hot topics in 5G and beyond use cases and applications in various hard propagation conditions.To avoid the overhead data in the pilot-aided schemes for channel estimation and equalization, the blind The associate editor coordinating the review of this manuscript and approving it for publication was Walid Al-Hussaibi .
approach has been widely investigated to reduce the ISI due to multipath propagation.
Moreover, the PANL can be mitigated in two ways: linearization techniques at the transmitter side and postdistortion approach at the receiver side.In the literature, various linearization methods are proposed such as feedforward linearization [1], feedback linearization [2] and the digital pre-distortion approach.The basic concept of the pre-distortion approach is to add a distorter at the transmitter side, based on an approximation of the PA inverse function.The main challenge provided by the digital pre-distortion approach is its high computational complexity that requires more energy and resources, leading to its limited use on the base stations side.To overcome the pre-distortion challenges, post-distortion has been widely investigated.It manages the PANL compensation on the receiver side.This approach is proposed for millimeter wave communication systems in 5G and beyond [3].In [3], two neural networks (NN) are integrated, the first for ISI mitigation and the second for PANL avoidance purposes.Authors in [4] propose a deep learning-based receiver with PA.In [5] a Volterra series-based convolution neural network (CNN) is suggested to mitigate PANL.In the literature, several studies based on post-compensation have been carried out [6], [7].In all these digital post-distortion-based solutions discussed above, the two coupled distortions: linear ISI and PANL, are addressed separately with non-blind training.In addition, we assume a single carrier transmission system where the spectrum emission constraints are satisfied.
In wireless communication, a multi-path channel introduces ISI.Blind equalization is one of the techniques applied to mitigate ISI, it avoids bandwidth waste resulting from training data.In the literature, various blind equalizers based on high-order statistical (HOS) properties are proposed as the constant modulus algorithm (CMA) [8] and the multi-modulus algorithm (MMA) [9].Other blind equalizers based on probability density fitting are also developed as the stochastic quadratic distance (SQD) [10] and the muti-modulus stochastic quadratic distance (MSQD-ℓ p ) [11].PDF-based equalizers that exploit the full range of signal distribution surpass equalizers based on HOS properties that exploit a part of signal features [11].For this reason, we are interested in PDF-based criteria in this paper.
In the last decade, the neural network has been widely operated in signal processing such as channel estimation and equalization.In the literature, several works were investigated to build blind neural equalizers like the neural network constant modulus algorithm (NNCMA) [12] and the neural network multi-modulus algorithm (NNMMA) [13].These algorithms outperform their linear versions: CMA and MMA.In the same way, we have previously implemented the SQD and MSQD-ℓ p PDF based criteria with a neural network [14] and have shown that using a neural network for these criteria outperforms their linear versions.
Numerous neural network architectures were considered for blind equalization.We can mention, the feedforward equalizer (FFE) [12], the feedforward with decision feedback equalizer (FFE-DF) [15], the recurrent neural network equalizer (RNNE) [16] and variational autoencoders [17].All these previous architectures process the real and imaginary parts of the signal separately, thus ignoring the correlation between the real and imaginary parts.In addition, the CV-NN [12] processes the real and imaginary parts together, preserving the correlation between them and consequently leading to a robust convergence.
In some communication scenarios, the receiver has no information on the transmission channel and the used constellation.Thus a GBE is used to cancel ISI.After equalization, and before taking the final decision and exploiting the received data, the receiver classifies the GBE output by an AMC algorithm to reach the transmitted constellation.
In the state of the art, some works consider jointly GBE and AMC.We can cite [18] where a multi-branch architecture is used, each branch is linked to a specific constellation.The branch provides the smallest error is considered.Other approaches use GBE based on CMA and MSQD-lp algorithms [19], [20].They use this criterion by reading with the quadratic phase shift keying (QPSK) constellation to equalize any constellation belonging to the phase shift keying modulation (PSK) or the quadrature amplitude modulation (QAM).The resulting constellation is scaled to ensure the targeted transmit power.All the cited works proposed a linear GBE followed by an AMC process.
In [21], we have implemented a joint neural GBE using CV-NN, and AMC based on a KNN classifier with the fourth-order cumulants which deal with quadrature amplitude modulation (QAM) [22], [23].
Moreover, since a PA has been included in a major part of the communication systems, in this paper we focus on its effect.In the literature, there are many PA models such as the polynomial model [24], [25], Rapp's model [26], [27] and Saleh's model [28].The nonlinearity behavior of these PA models is usually characterized by their amplitude modulation-amplitude modulation (AM-AM) and amplitude modulation-phase modulation (AM-PM) conversions.In this paper, we consider the modified Rapp model [29], which has both AM-AM and AM-PM conversion characteristics.
Employed orthogonal frequency division multiplexing (OFDM) is used in communication systems such as 5G and beyond [3].However, there are other systems that still use the single carrier communication system such as maritime communication, satellite communication, and the Internet of Things (IoT) technologies.The work proposed here considers single input single output (SISO) communications and we are currently working on its extension to MIMO systems.
In this work, we propose a fully blind Deep Learning(DL)based signal receiver with two sequential CV-NNs banking on the probability density fitting (PDF) criterion.The first one mitigates linear ISI and the second one compensates PANL.We also propose a generic blind DL-based signal receiver that mitigates the ISI by the neural network GBE (NNGBE) and the PANL by the neural network GBPDC (NNGBPDC).
Accordingly, the main contributions of this paper are resumed as follows: • Joint simultaneously blind channel equalization, and blind post-distortion compensation.
• Joint simultaneously generic blind channel equalization, and generic blind post-distortion compensation.To our knowledge, this is the first time that the PDC is processed blindly.
The rest of this paper is organized as follows.Section II describes the system and signal models.Section III introduces the blind DL-based signal receiver.The generic blind DL-based signal receiver is described in section IV.
Simulation results in a complete transmitter/receiver chain are discussed in section V. Finally, Section VI concludes this work.

II. SYSTEM MODEL A. SIGNAL MODEL
In the rest of this paper, we consider the following parameters and notations.We note that lowercase letters, e.g.x, and bold lowercase letters, e.g.x, denote scalars and vectors respectively.ℜ(.) or (.) r denotes the real part of a complex number, and ℑ(.) or (.) i denotes the imaginary part of a complex number.* denotes the conjugate operator.
Our base-band communication system is detailed in Fig. 1.
Transmission system base-band model with a power amplifier, and DL-based signal receiver.
• s(n) n∈Z is a normalized M-QAM sequence of independent and identically distributed (i.i.d) complex symbols, • x(n) is the PA input and α is a multiplicative factor to adjust the average power of the signal at the PA input, • i 1 (n) and o 1 (n) are the input and the output of the first CV-NN equalizer at a given time n, respectively, where G 1 is the first CV-NN three layers global function, ] T is a vector of L 1 complex data samples applied at its input layer.w 1 is the matrix of synaptic weights between the input and hidden layers and w 2 is the matrix of synaptic weights between the hidden and output layers.
• i 2 (n) and o 2 (n) are the input and the output of the second CV-NN at a given time n, respectively: where G 2 is the second CV-NN three layers global function, i 2 (n) = [p 1 (n), . . ., p L 2 +1 (n)] is a vector of L 2 complex data samples applied at its input layer, with . u 1 is the matrix of synaptic weights between the input and hidden layers and u 2 is the matrix of synaptic weights between the hidden and output layers.

B. CV-NN MODEL
The first CV-NN structure is shown in Fig. 2. In the sequel, we assume N k neurons in the k th layer and we denote by φ k j and ν k+1 j is the input and the output of the j th neuron, such that: where ν k i is the k th layer input, w k ij is the weight between the i th neuron in the k th layer and the j th neuron in the (k + 1) th layer, θ k j and f k (•) are the k th layer bias and activation function, respectively.The CV-NN is trained using the complex backpropagation (CBP) algorithm [30].
The second CV-NN has the same structure as the first CV-NN with the appropriate inputs defined in the previous subsection.

C. POWER AMPLIFIER MODEL
In this work, we assume the memoryless nonlinearity introduced by the modified Rapp's PA model.This PA is commonly used to model solid-state power amplifiers (SSPAs) [31] and exhibits both the AM-AM and AM-PM conversion characteristics.The signal at the output of the PA model can be written as follows: where ρ(n) and ϕ(n) are respectively the input signal (x(n)) modulus and phase.F a (•) is the AM/AM conversion function, Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
and P a (•) is the AM/PM conversion function, expressed, for the modified Rapp model, as follows: where g is the linear gain, V sat is the saturation voltage and p is a smoothness factor that controls the transition from the linear region to the saturation one.k, u and q respectively represent the adjustment parameters.In this paper, we assume that V sat = 1.9, g = 16, k = −345, u = 0.17, p = 1.1 and q = 4 [26], [32].This PA is applied for modeling sub−6GHz communications for 4G and 5G systems [3].In this paper, we ignore the memory effects because the PA response is assumed to be constant over the signal band [33].
The PA nonlinear conversion characteristics can produce distortions in the constellation scheme, as well as spectral regrowth, degrading the system performance [4].In practice, to reduce the nonlinearity effects, the PA is operated at a given input back-off (IBO) level from its saturation point [3].In the logarithmic scale, the IBO is defined by IBO = 10 log 10 V 2 sat P x , where P x is the average power of the PA input signal x(n) = α s(n).We remind that α is a scaling factor adjusted to reach the targeted IBO value.

III. BLIND DL-BASED SIGNAL RECEIVER
Communications systems integrate usually power amplifiers, whose non-linear characteristics may affect transmission quality.The received signal is impacted by two coupled effects: ISI and PANL.ISI is often mitigated by the receiver, but the PANL is compensated in two ways: either by pre-distortion at the transmitter or with post-distortion at the receiver side.In this section, we will detail the blind DL-based signal receiver that we propose.It has the advantage to equalize the wireless transmission channel and compensate the PANL.
The receiver includes two sequential CV-NNs as shown in Fig. 3.The first one aims to mitigate ISI and the second one aims to compensate PANL.The Two sequential CV-NNs are trained by employing the MSQD-ℓ p algorithm [11].Specially for p = 2, the MSQD-ℓ 2 has the following cost function: where N s is the number of complex symbols in the considered constellation and K σ 0 is a Gaussian kernel with zero mean and variance σ 0 which is referred to kernel width.o and s are the equalized output and the constellation symbols, respectively.The blind equalization criterion based on PDF tries to estimate the distribution of the transmitted data at the receiver side.It minimizes the distance between the signal PDF at the equalizer output and the emitted constellation during iterations.

A. ISI CANCELLATION (CV-NN 1)
In the first stage, CV-NN 1 is trained according to (7) in order to mitigate ISI.CBP was investigated for the training process and we consider the stochastic gradient descent (SGD) algorithm to update the network weights, such that: The gradient is calculated as in [14].For simplification purposes, we introduce two auxiliary parameters Q r and Q i which are defined as follows: and Further, the weights of the output layer are updated as the following: where ϕ o is the input of the output layer.I pj is the output of the hidden layer.f o (.) is the activation function of the output layer.f o ′ (.) is the derivative of the activation functions of the output layer.The weights of the hidden layer are updated as follows: where ϕ h is the input of the hidden layer and y pi is the input of the network.f h ′ j (.) is the hidden layer derivative activation function for the j th neuron.

B. PANL COMPENSATION (CV-NN 2)
After training CV-NN1, we obtain its optimal parameters.Then, the training of CV-NN 2 is performed, and where the training process is the same as the CV-NN1 (CBP, SGD) one.The input of CV-NN2 is a polynomial constructed from the output of CV-NN1 as described in equation (2).
Once the learning is done for both networks and the optimal parameters are procured, both models are performed online to cancel ISI in the first stage and compensate PANL in the second stage.

IV. GENERIC BLIND DL-BASED SIGNAL RECEIVER
We now assume that the receiver has no information about the used constellation.So, it becomes a generic receiver for different possible M-QAM constellations and it is built from two sequential CVNNs with an AMC block as shown in Fig. 4. The first CV-NN mitigates ISI, and the second one compensates PANL.We have trained the first CV-NN with a new multi-criteria generic cost function as a sum of several variations of equation (7), one for each constellation order multiplied by an updated penalty factor.So, the criterion will be: where C is the number of the constellation orders.
The penalty factor α m in ( 12) is updated in each iteration according to equation ( 13) such that we reach, over the iterations, the transmitted constellation cost function: where class m is the number of times where the m th constellation is successful in the classification result after the classification step.At the end of the CV-NN 1 training process, (13) will converge to a specific cost function that deals with the classified constellation.This cost function will be used for the CV-NN 2 training.

A. ISI CANCELLATION (CV-NN 1)
In the first step, CV-NN 1 is trained according to (12) in order to mitigate ISI.CBP was investigated for the training process and we consider the stochastic gradient descent (SGD) algorithm to update the network weights, such that: In the following, we will concentrate on the gradient of (12).To do so, we use the gradient of ( 7), since the gradient of ( 12) is a sum of the ( 7) gradient one for each constellation order.
The weights of the output layer are updated as: where I pj is the output of the hidden layer and δ o p m is calculated as the same in subsection III-A for each m th constellation order.
Similarly, the weights of the hidden layer are updated as: where y pi is the input vector of the CV-NN and δ h pj m is calculated as the same in subsection III-A for each m th constellation order at the j th neuron.
In order to update the cost function, each equalized symbol is classified using the KNN algorithm with fourth-order cumulant [34], [35] as features.So that we execute jointly the equalization and the AMC.The expressions of the p th order cumulant and moment are given by: C pq = cum(x p−q (x * ) q ) and where E[•] is the expectation operator.
In particular, we will consider: as they are more suitable for M-QAM modulations [35].
A baseline containing C 40 and C 42 values for noisy symbols according to various values of signal-to-noise ratio (SNR) and belonging to {16, 32, 64, 128, 256}-QAM modulations is prepared.
The classifier follows the following steps: • Calculate S 1 = C 40 + C 42 the sum of the last 1000 equalized symbols.
• Calculate the Euclidean distances between S 1 and each sum in the reference base.
• A set of the k nearest neighbors from the baseline is created.
104758 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
• Select the mostly repeated order in the previous, as each neighbor matches a specific constellation order.Finally, according to the AMC, we update the penalty factors following (13) and update (12).

B. PANL COMPENSATION (CV-NN 2)
Once CV-NN 1 is trained and converging, we continue training CV-NN 2 with the resulting cost function from CV-NN 1.The output of CV-NN 1 with a polynomial form is the CV-NN 2 input as described in equation ( 2) and we use the same training process (CBP, SGD).
As in the blind approach, once the learning and classification phases are done for both networks and the optimal parameters are obtained, both models are performed online to mitigate ISI and classify the used constellation in the first step, and compensate PANL in the second step.
These parameters correspond to a mobile speed of 30 meter per second and a carrier frequency of 2 gigahertz.
We have initialized the two matrices of synaptic weights w o [L o , L h ] and w h [L h , L i ] with small values except for w o [(L o + 1)/2, (L h + 1)/2] and w h [(L h + 1)/2, (L i + 1)/2] that were set to 1.5.
The total number of symbols used during the training step is 70000.An important factor for convergence during the training phase is the learning rate µ.It is fixed to 10 −3 /P y , where P y is the received signal power.All the compared algorithms in this paper have the same convergence speed and in all figures, we draw the BER values after the convergence.
We have taken CV-NN 1 trained in nonblind mode followed by analytic compensation of PANL as a DL-based signal receiver benchmark.To compute the analytical compensation, we use the Newton's approximation algorithm such in [36] to numerically calculate the value of the inverse of the AM/AM conversion.We then use this calculated value to obtain the AM/PM conversion to be extracted from the phase of the signal at the output of CV-NN 1.Thus, we have simulated the DL-based signal receiver composed of CV-NN 1 and CV-NN 2 using 16-QAM and 32-QAM signal constellations in both blind and generic versions.The different BER figures presented in this paper are taken with respect to various signal-to-noise ratio (SNR) values [37], [38].
For nonlinear effects, We have implemented the modified Rapp model described in Section II-C.The polynomial non-linearity at the input of CV-NN 2 L 2 is fixed to 5 containing the even and the odd orders.The computational complexity order for both CV-NN 1 and CV-NN 2 is the same.It is a linear function of neuron number for each network layer.
Computation carried for CV-NN 1 and CV-NN 2 are summarized in table 2. A. BLIND DL-BASED SIGNAL RECEIVER Fig. 5 confirms that our blind DL-based signal receiver with a 16-QAM received signal and a radio channel, outperforms the blind ISI cancellation without PANL compensation, especially for higher SNR values.The BER Performance is also affected by the IBO.In fact, the higher the IBO is, i.e. the PA is away from its saturation point, the more performance increases.Thus, our blind DL-based signal receiver built by two coupled CV-NNs solutions guarantees performance close to the analytic one.
We have also tested our receiver with a 32-QAM received signal and a time-varying vehicular channel.Fig. 6 describes different blind approaches in the case of the vehicular time-varying channel h v .We can observe the robustness of our approach even with a time-varying channel and different constellation orders.

B. GENERIC BLIND DL-BASED SIGNAL RECEIVER
We trained CV-NN 1 with the multi-criteria cost function.For each equalized symbol, we apply the KNN algorithm for classification and finally, we update the cost function to reach one that matches the classified constellation.Once the CV-NN 1 is trained, we train CV-NN 2 with the resulting cost function obtained after the convergence of CV-NN1 without applying the AMC.
The sub-figures in Fig. 7 are simulated with a 16-QAM signal and a radio channel in a generic mode.The sub-figures in Fig. 8 are simulated with a 32-QAM signal and the time-varying vehicular channel in a generic mode.The obtained results confirm the performance of our contribution in a generic mode with different constellations orders and channels propagation.We can see that the constellation sub-clusters are not distributed in the right places.Also, there are overlaps between the decision areas corresponding to the different symbols.Indeed CV-NN 1 has mitigated only ISI and not PANL, which will lead to higher BER values.Fig. 9 describes different generic approaches simulated with a 16-QM signal and the digital radio channel h r .We can observe a slight performance degradation for all algorithms compared to Fig. 5.This was expected since the receiver has no idea about the transmitted modulation and the cost function used for this case is generic.
In Fig. 10, we present the performance of the generic DL-based signal receiver with a 32-QAM signal and the vehicular time-varying channel h v .We noticed an interesting BER performance and robust convergence despite channel time variation during transmission.

VI. CONCLUSION
This paper presents two contributions.The first one jointly and blindly corrects the effects of inter-symbol interference introduced by the propagation channel with the non-linearities introduced by power amplifiers without memory and presenting both amplitude and phase conversions.The second one jointly and generic corrects the same effects.Each of the two proposed non-linear receivers is a cascade of two neural networks, the first of which corrects the ISI and the second compensates the non-linearities.For the blind receiver, the two complex-valued neural networks are trained by employing the MSQD-ℓ 2 criterion, and for the generic receiver the first CV-NN is trained using an updated multi-criteria cost function, and the second CV-NN is trained using the resulting cost function from the CV-NN 1.The numerical results show a good performance in terms of BER with a stationary channel and a time-varying vehicular channel.The performance achieved by the DL-based signal receiver motivates us to extend our work to multiple inputs multiple outputs (MIMO) communication systems in the presence of power amplifiers exhibiting memory effects.

FIGURE 6 .
FIGURE 6. BER versus SNR and IBO using h v in the case of a 32-QAM signal in a blind mode.

Fig. 7 (
a) and Fig. 8(a) show the data source to be transmitted, Fig.7(b) and Fig. 8(b) illustrate the data after the application of PA, Fig. 7(c) and Fig. 8(c) draw the received data at the channel output, these data are the receiver input and Fig. 7 (d) and Fig. 8 (d) plot the CV-NN 1 output signal constellation.

FIGURE 7 .
FIGURE 7. Different constellation for IBO = 4dB and SNR = 20dB using h r with 16-QAM signal in generic mode.

FIGURE 8 .
FIGURE 8. Different constellation for IBO = 4dB and SNR = 20dB using h v with 32-QAM signal in generic mode.

FIGURE 9 .
FIGURE 9. BER versus SNR and IBO using h r in the case of a 16-QAM signal in a generic mode.

Fig. 7 (
Fig. 7 (e) and Fig. 8 (e) plot CV-NN 2 output signal constellation.It can be noticed that the constellation sub-clusters are almost distributed at the proper positions.In addition, the decision areas corresponding to the different symbols are well separated.This is reflected in low BER values.This performance improvement is due to the ISI mitigation and PANL compensation by CV-NN 1 and CV-NN 2, respectively.

FIGURE 10 .
FIGURE 10.BER versus SNR and IBO using h v in the case of a 32-QAM signal in a generic mode.

TABLE 2 .
Computational complexity.FIGURE 5. BER versus SNR and IBO using h r in the case of a 16-QAM signal in a blind mode.