High-Performance Noise Whitening Filters for Linear and Nonlinear Colored Noise Suppression in IM/DD System With Bandwidth Limitation

In this paper, we propose three high-performance noise whitening filters (NWF) for linear and nonlinear colored noise, including a scale-enhanced NWF (SE-NWF), an inverse-average NWF (IA-NWF), and an inverse-average nonlinear NWF (IA-NNWF). After obtaining a traditional linear NWF based on auto-regressive modeling, the SE-NWF incorporates an amplification factor to increase the spectral range for stronger noise-whitening ability. To further improve linear and nonlinear noise-whitening ability, the novel IA-NWF and IA-NNWF are presented, which are based on the fundamental tenet that the inversion of the noise spectrum and the extrapolation of this process through the application of linear and nonlinear finite impulse response filters, respectively. The proposed schemes are evaluated in a C-band 61-GBaud intensity modulation and direct detection (IM/DD) optical on-off keying (OOK) and 4-level pulse amplitude modulation (PAM-4) systems. In the OOK transmission with a 3.7-GHz 3-dB bandwidth over an 80-km standard single-mode fiber (SSMF), the results demonstrate that the 8-tap SE-NWF and 8-tap IA-NWF exhibit comparable performance to the 19-tap and 21-tap conventional NWFs, respectively. With the same memory length, the SE-NWF, IA-NWF, and IA-NNWF can achieve approximate improvements of 2.8 dB, 3.0 dB, and 3.2 dB on receiver sensitivity, respectively, in comparison to the conventional NWF under a 20% forward error correction (FEC) threshold. In the PAM-4 transmission over a 2-km SSMF, compared to the conventional NWF of equivalent complexity, the 2-tap proposed schemes achieve approximately 1.6 dB and 4.2 dB improvements in receiver sensitivity at the 20% and 7% FEC thresholds, respectively.