Hybrid Probabilistic-Geometric Shaped Constellations to Combat Fiber Non-Linearity

We propose a hybrid probabilistic-geometric constellation shaping method for optical fiber communication systems that is non-linearity tolerant and compatible with probabilistic fold shaping (PFS) architecture. To do so, the impact of non-linear interference noise (NLIN) in the shaping process is considered. This is a challenging process because NLIN is itself constellation dependent. This study copes with this issue by developing a deep learning-based shaping method that takes the enhanced Gaussian noise (EGN) model of fiber into account to model NLIN. Our hybrid shaping scheme maximizes the generalized mutual information (GMI) rate of the optical system while the impacts of shaping on the NLIN power are considered. Our results show that the proposed hybrid scheme results in significant reach improvements and outperforms recent hybrid shaping methods. Interestingly, our hybrid shaping scheme suggests distributions that are much different from the traditional Maxwell-Boltzmann distributions.