Dual-Functional MIMO Beamforming Optimization for RIS-Aided Integrated Sensing and Communication

Aiming at providing wireless communication systems with environment-perceptive capacity, emerging integrated sensing and communication (ISAC) technologies face multiple difficulties, especially in balancing the performance trade-off between the communication and radar functions. In this paper, we introduce a reconfigurable intelligent surface (RIS) to assist both data transmission and target detection in a dual-functional ISAC system. To formulate a general optimization framework, diverse communication performance metrics have been taken into account including famous capacity maximization and mean-squared error (MSE) minimization. Whereas the target detection process is modeled as a general likelihood ratio test (GLRT) due to the practical limitations, and the monotonicity of the corresponding detection probability is proved. For the single-user and single-target (SUST) scenario, the minimum transmit power for sensing has been revealed. By exploiting the optimal conditions, we validate that the optimal BS satisfies the maximum power allocation criterion and derive the optimal BS precoder in a semi-closed form. Moreover, an alternating direction method of multipliers (ADMM) based RIS design is proposed to address the non-convex radar constraint. For the sake of enhancing computational efficiency, a low-complexity RIS design is also developed based on the manifold optimization theory. Furthermore, the ISAC transceiver design for the multiple-users and multiple-targets (MUMT) scenario is also investigated, where a zero-forcing (ZF) radar receiver is adopted to cancel the interference signals from different targets. Then optimal BS precoder is derived under the maximum power allocation scheme, and the RIS phase shifts can be optimized by extending the proposed ADMM-based RIS design algorithm. Finally, the ISAC transceiver design with imperfect in-band full-duplex transceivers is also discussed and two radar receive beamformer designs have been proposed to mitigate the performance loss. Numerical simulation results verify the convergence and superior communication/sensing performance of our proposed transceiver designs.