NN-Assisted Message Passing Based Bayesian Joint DOA Estimation and Signal Detection for ISAC Systems With Hardware Imperfections

This work investigates communication signal detection and direction of arrival (DOA) estimation for an integrated sensing and communications (ISAC) system with multiple hardware imperfections, including power amplifier nonlinearity, in-phase and quadrature phase imbalance, and phase-gain error (PGE). Conventional signal processing techniques struggle with the complex nonlinearities arising from these imperfections. Recently, deep neural networks (DNNs) have been employed to mitigate hardware impairments; however, they require a substantial number of pilot signals for training, leading to significant overhead, making them impractical in many applications. In this work, we design a signal flow inspired neural network (NN) to characterize the nonlinear ISAC system. Then, we propose a Bayesian method to jointly estimate the parameters of the PGE, the communication signals, and the DOAs by developing a message-passing inference algorithm based on the NN. Extensive simulation results demonstrate that the proposed method provides efficient and robust signal detection and DOA estimation performance under PGE, and significantly outperforms state-of-the-art ones.