Machine Learning Techniques in Optical Communication

Machine learning techniques relevant for nonlinearity mitigation, carrier recovery, and nanoscale device characterization are reviewed and employed. Markov Chain Monte Carlo in combination with Bayesian filtering is employed within the nonlinear state-space framework and demonstrated for parameter estimation. It is shown that the time-varying effects of cross-phase modulation (XPM) induced polarization scattering and phase noise can be formulated within the nonlinear state-space model (SSM). This allows for tracking and compensation of the XPM induced impairments by employing approximate stochastic filtering methods such as extended Kalman or particle filtering. The achievable gains are dependent on the autocorrelation (AC) function properties of the impairments under consideration which is strongly dependent on the transmissions scenario. The gain of the compensation method are therefore investigated by varying the parameters of the AC function describing XPM-induced polarization scattering and phase noise. It is shown that an increase in the nonlinear tolerance of more than 2 dB is achievable for 32 Gbaud QPSK and 16-quadratic-amplitude modulation (QAM). It is also reviewed how laser rate equations can be formulated within the nonlinear state-space framework which allows for tracking of nonLorentzian laser phase noise lineshapes. It is experimentally demonstrated for 28 Gbaud 16-QAM signals that if the laser phase noise shape strongly deviates from the Lorentzian, phase noise tracking algorithms employing rate equation-based SSM result in a significant performance improvement (
$>$
8 dB) compared to traditional approaches using digital phase-locked loop. Finally, Gaussian mixture model is reviewed and employed for nonlinear phase noise compensation and characterization of nanoscale devices structure variations.