End-to-End Deep Learning for Long-haul Fiber Transmission Using Differentiable Surrogate Channel

Recently, end-to-end deep learning (E2EDL) has been proposed for communication systems to improve the overall performance of systems. In the domain of optical fiber communication systems, the theory of E2EDL and the backpropagation (BP) blocking problem have not been reported in detail. Many works focus on the simple channel with an explicit model (e.g., self-phase modulation-only or dispersion-only fiber channel), while long-haul fiber transmission with both dispersion and nonlinearity has not been discussed because of the BP blocking problem. This paper analyzes why end-to-end deep learning can optimize the transmitter effectively in theory and derives the mean square error (MSE) loss for E2EDL, which is suitable for practice training. We discuss the BP problem mathematically and utilize a gradient estimation method called differentiable surrogate channel (DSC) to address the BP blocking problem. For the first time, the E2EDL is applied to a long-haul fiber channel to optimize the transmitter by using DSC-based gradient estimation. The simulation results demonstrate the mutual information (MI) and the generalized mutual information (GMI) increment with the training process, verifying the effectiveness of the E2EDL in long-haul fiber transmission. In addition, the proposed algorithm is applied to geometric constellations shaping (GCS) for long-haul transmission to explore the performance boundary of GCS. After GCS-based optical fiber E2EDL, the Q factor and the GMI can be improved around 0.25 dB and 0.11 bits/sym at 1200 km for 48 GBaud 64-ary dual-polarization signal compared with the square QAM signal, respectively. In addition, the trained 16-ary, 32-ary, and 64-ary GCSs are tested from 800 km to 2800 km, 800 km to 2000 km, and 400 km to 1200 km, respectively. We believe the proposed E2EDL with DSC can achieve a tight lower bound of GCS performance limitation and open the door to transmitter optimization in the practical long-haul fiber transmission.