Gradient Boosting for Nonlinear Equalization in Optical Transmission Systems

The performance of modern optical communication systems is a bottleneck of modern technology, and to enhance this performance, we must mitigate the linear and nonlinear distortions that occur during signal propagation through optical fibers. There have been numerous studies devoted to this problem, with ideas for digital signal processing (DSP) algorithms for fiber nonlinear equalization. In recent years, the advantages of machine learning (ML) techniques have inspired researchers to expand these advantages for nonlinear equalization. Several works have demonstrated the ability of smart equalizers based on neural networks (NN) to “understand” system parameters and predict nonlinear distortions [1]. The goal of this study was to explore the use of Gradient Boosting (GB) technique for nonlinear equalization in optical transmission systems. GB is an ensemble learning method that operates by constructing a multitude of decision trees. In this case, GB is used as a regression model, which returns the average prediction of the individual trees for nonlinear shift. In practical implementation, the use of GB can be implemented through the use of pretrained switchers on a field-programmable gate array (FPGA) [2].