Experimental Investigation of Deep Learning for Digital Signal Processing in Short Reach Optical Fiber Communications

We investigate methods for experimental performance enhancement of auto-encoders based on a recurrent neural network (RNN) for communication over dispersive nonlinear channels. In particular, our focus is on the recently proposed sliding window bidirectional RNN (SBRNN) optical fiber auto-encoder. We show that adjusting the processing window in the sequence estimation algorithm at the receiver improves the reach of simple systems trained on a channel model and applied "as is" to the transmission link. Moreover, the collected experimental data was used to optimize the receiver neural network parameters, allowing to transmit 42Gb/s with bit-error rate (BER) below the 6.7% hard-decision forward error correction threshold at distances up to 70 km as well as 84 Gb/s at 20 km. The investigation of digital signal processing (DSP) optimized on experimental data is extended to pulse amplitude modulation with receivers performing sliding window sequence estimation using a feed-forward or a recurrent neural network as well as classical nonlinear Volterra equalization. Our results show that, for fixed algorithm memory, the DSP based on deep learning achieves an improved BER performance, allowing to increase the reach of the system.