Nonlinearity-Robust IM/DD THz Communication System via Two-Stage Deep Learning Equalizer

Linear and nonlinear impairments restrict the transmission performance of high-speed terahertz (THz) communication systems. To improve transmission performance, we propose a two-stage nonlinear equalizer (NLE). In the first stage, a memory-controlled long short-term memory (LSTM) neural network learns channel nonlinearity and compensates for it through nonlinear waveform regression. In the second stage, a low-complexity deep random forest (RF) network identifies nonlinear boundaries among individual QAM symbols and adjusts the standard hard decision thresholds of the QAM demodulator to align with the distribution of received symbols. This study experimentally validates the proposed two-stage NLE on a dual-channel THz-over-fiber transmission system using an intensity modulation and direct detection (IM/DD) scheme, achieving a successful 20 Gbps line rate up to a 4.5-meter wireless link at both 125/300 GHz frequency bands. The proposed scheme outperforms a Volterra nonlinear equalizer in all tested scenarios, surpassing a linear equalizer (LE) by reducing the bit error rate (BER) from
$2.47 \times 10^{-3}$
to
$2.61 \times 10^{-4}$
in the 300 GHz link and from
$3.42 \times 10^{-3}$
to
$5.64 \times 10^{-4}$
in the 125 GHz link.