Hybrid GMM-GS Message Passing Detection for OTFS Modulation

Orthogonal Time Frequency Space (OTFS) modulation can effectively combat the Doppler effects and exploit the time-frequency resources. The sparsity of the OTFS channel matrix in delay-Doppler domain plays a crucial role in message passing detection algorithms. This paper presents a hybrid detection approach that combines Gaussian mixture model (GMM) with Gaussian approximation (GS). According to the path gain, GMM is employed to deal with significant paths and GS for less significant paths. The proposed hybrid method achieves a balance between computational complexity and performance by indirectly computing messages and introducing a damping factor. Simulation results show that the proposed method exhibits noteworthy Bit Error Rate (BER) performance and reduces the computational complexity associated with GMM.