ML-Assisted Particle Swarm Optimization of a Perturbation-Based Model for Nonlinearity Compensation in Optical Transmission Systems

Nonlinear signal distortions are one of the primary factors limiting the capacity and reach of optical transmission systems. Currently, several approaches exist for compensating nonlinear distortions, but for practical implementation, algorithms must be simultaneously accurate, fast, and robust against various interferences. One established approach involves applying perturbation theory methods to the nonlinear Schrödinger equation, which enables the determination of the relation between transmitted and received symbols. In most studies, gradient methods are used to find perturbation coefficients by minimizing the mean squared error between symbols. However, the main parameter characterizing the quality of information transmission is the bit error rate. We propose a modification of the conventional perturbation-based approach for fiber nonlinearity compensation in the form of a two-stage scheme for calculating perturbation coefficients. In the first stage, the coefficients are computed using a single 1D convolutional layer by minimizing the mean squared error. In the second stage, the obtained solution is used as an initial approximation for minimizing the bit error rate using the particle swarm optimization method. In numerical experiments, using the nonlinearity compensation algorithm based on the proposed scheme, we achieved a 0.8 dB gain in the signal-to-noise ratio for a 16QAM 20 × 100 km link with a channel net rate of 222 Gbit/s and demonstrated improved accuracy compared to the single-stage scheme. We estimated computational complexity of the algorithm and demonstrated the relation between its complexity and accuracy. Additionally, we developed a method for learning perturbation coefficients without relying on ideal symbols from the transmitter, instead using the received symbols after hard decision detection.