Quantization of Recurrent Neural Network for Low-Complexity High-Speed IM/DD System Equalization Based on Neuron Clustering

Neural networks (NNs) are widely employed as effective equalizers in intensity-modulated direct-detection (IM/DD) links due to their excellent ability in dealing with nonlinear channel impairments. However, the complexity concern impedes the real-time application of NN-based receivers. To address this issue, we propose mixed-precision quantization of recurrent NN (RNN)-based equalizers in a 100-Gb/s 15-km C-band IM/DD system, which saves about 73.3% and 22.4% memory compared with traditional floating-point-based and fixed-precision quantized RNN. A simple and effective neuron clustering approach is proposed to realize mixed-precision quantization of RNN without degrading system performance.