Model-Free End-to-End Deep Learning of Joint Geometric and Probabilistic Shaping for Optical Fiber Communication in IM/DD System

The application of neural network (NN)-based autoencoders in end-to-end (E2E) optimized communication systems has significantly improved performance. Autoencoders offer automated transmission signal optimization capabilities, notably by integrating geometric and probabilistic constellation shaping (GPS). However, the practical application of GPS-AE in E2E optimization faces challenges due to its reliance on an accurate, robust, and differentiable channel model—a requirement often difficult to meet. To address this challenge, we introduce a model-free learning algorithm that enables bit-wise GPS-AE training directly on real communication system channels. Our method incorporates a trainable equalizer into the E2E learning process to compensate channel impairments, allowing the GPS-AE decoder to handle only adaptive noise. Using this technology, the GPS-AE neural transceiver refines constellation distributions to enhance communication performance over real channels. Experimental results show that our solution exhibits substantial net bitrate advancements, delivering up to 349.2 Gb/s over 0.5-km SSMF transmissions—exceeding the Maxwell-Boltzmann distributed PS-QAM baseline by 51.4 Gb/s in IM/DD optical communication system. These results highlight the potential optimization capabilities of our proposed model-free GPS-AE solution in future optical fiber communication systems.