Autoencoder Learning of Nonlinear Constellation Shape for Fiber-Wireless Convergence System

We propose and experimentally demonstrate a novel nonlinear constellation shape auto-optimization method with a complex-valued 2D-ANN equalizer. Up to 70% lower BER compared with the conventional format is achieved at 50 Gbps in fiber-MMW system.