Neural Network Aided FTN Transmission Over Doubly Selective Fading Channels: Waveform Learning, Channel Estimation and Data Detection

In this paper, we address the key challenges of faster-than-Nyquist signaling (FTNS) in doubly selective fading channels (DSCs) by proposing an end-to-end architecture that jointly optimizes FTN waveform, data detector, and channel estimator. Unlike traditional model-driven approaches, our solution introduces a data-driven optimization strategy, enhancing its ability to optimize parameters on a larger scale. In the proposed architecture, constellation geometry, precoder, and pilot sequences are considered as optimizable variables. Specifically, with the goal of maximizing spectral efficiency (SE), the constellation geometry and precoder are co-trained with a neural network (NN)-based data detector, showing superior performance in mitigating inter-symbol interference (ISI) caused by FTNS and channel spreading. Concurrently, the pilot sequences are optimized alongside the channel estimator to improve the estimation accuracy of CIR at pilot blocks. To track rapidly time-varying CIRs, we apply an attention mechanism to the channel estimator that efficiently leverages the channel's temporal correlation through dynamically assigning weights to the pilot blocks to accurately predict the CIRs for each data block. Simulation results show that our scheme achieves a signal to noise ratio gain of more than 6 dB under a maximum Doppler shift of 10 kHz compared with the state-of-the-art linear pre-equalized FTN system with 64-ary quadrature amplitude modulation.