Recurrent Neural Network Soft-Demapping for Nonlinear ISI in 800Gbit/s DWDM Coherent Optical Transmissions

High speed optical transmission systems suffer from intersymbol interference (ISI) and colored noise induced by nonlinear bandwidth limited optical and electrical components. As a countermeasure, this article investigates deep neural network soft-demappers. In particular, we propose a bidirectional recurrent neural network soft-demapper (BRNN-SD) and benchmarked its performance against a time delay neural network soft-demapper (TDNN-SD) and a reference digital signal processing (DSP) scheme consisting of a Volterra nonlinear equalizer accompanied by a symbol-spaced whitening filter and a BCJR detector. On coherent 92GBd dual polarization (DP)-32QAM back-to-back measurements, the proposed soft-demapper matches the performance of the reference DSP. In 800 Gb/s 96GBd DP-32QAM 32-channel dense wavelength division multiplexing (DWDM) transmission over a 600 km fiber link the proposed approach outperforms the reference DSP.