Optimization of the Non-Linearity Tolerant 4D Geometric Shaped Constellations for Optical Fiber Communication Systems Using Neural Networks

We propose a novel 4D geometric shaping (GS) method based on multilevel coding (MLC) for optical fiber communication systems that mitigates fiber non-linearity. To design non-linearity tolerant 4D constellations, non-linear interference noise (NLIN) and modulation dependency behavior of the fiber are considered during the optimization process. By doing so, the complexity of the 4D GS problem increases extremely, requiring novel optimization approaches to tackle this challenge. Hence, we build a novel deep neural network structure that can efficiently optimize the position of the 4D points. Moreover, we show that optimizing the geometry based on the well-known generalized mutual information rate is not efficient and results in a significant gap to the mutual information rate. To solve this issue, we take advantage of the concept of achievable information rates (AIRs) and multilevel coding. By optimizing the 4D constellations based on the proposed AIRs, our GS scheme provides higher rates than the traditional probabilistic shaping and the recent non-linearity tolerant GS methods.