MMSE-Driven Signal Constellation Scatterplot Using Neural Networks-Based Nonlinear Equalizers

This study investigates a phenomenon observed in signal constellation diagrams when using neural networks (NNs) based nonlinear equalizers optimized with the Minimum Mean Squared Error (MMSE) criterion. This phenomenon is characterized by a concentration of the symbols around the original constellation points, with some scattered along straight lines connecting neighboring points of the original constellation. We refer to this effect as “MMSE-driven signal constellation scatterplot” (MMSE-scatterplot). This phenomenon has harmful implications for subsequent signal processing, particularly in Soft-Decision (SD) Forward Error Correction (FEC) schemes, which require reliable soft information. Indeed, the MMSE-scatterplot behaves like a hard decision or denoiser, resulting in the removal of soft-information. In this paper, we explicitly relate the MMSE-scatterplot with a function named here Soft-Thresholding (STH). Additionally, to avoid the MMSE-scatterplot emergence on the equalized symbols, we propose the inclusion of the STH function as a nonlinear activation function after the NN during the training stage. This approach permits separating the equalization stage, performed by the NN, and the denoising stage, performed by the STH function, the latter giving rise to the MMSE-scatterplot. Consequently, to recover the equalized symbols in the evaluation stage, we remove the STH function, giving as a result a constellation diagram free of MMSE-scatterplot. To assess the effectiveness of this technique, we use a numerical setup DP-64QAM transmission system with 14 × 50 km of SSMF for various input signal powers. We also compare our results, in terms of bit error rate (BER) and Mutual Information (MI), with the obtained ones using an NN optimized with the recently proposed MSE-X loss function. Our results show that both NN+STH (using MSE) and NN (using MSE-X) efficiently permit to recover the equalized signal constellation free of MMSE-scatterplot, with good MI and with a slightly better BER using the NN+STH (MSE) than using the NN (MSE-X).