Physical Layer Security Enhancement for Beyond Diagonal RIS-Aided MIMO Communications

Beyond diagonal reconfigurable intelligent surface (BD-RIS) has demonstrated remarkable performance over conventional RIS through its unique flexible signal manipulation capability. This letter investigates physical layer security enhancement in a multiple-input multiple-output (MIMO) system aided by BD-RIS under eavesdropping channel. To achieve the maximal secrecy rate, we co-optimize the transmit covariance matrix and the BD-RIS unitary matrix. First, the projected gradient method refines the transmit beamforming to adhere to power constraints. Second, we propose an adaptive Riemannian conjugate gradient algorithm that incorporates low-order Taylor series expansions for geodesic approximation and hybrid conjugate parameters for co-optimization, ensuring the unitary constraint of the BD-RIS matrix. Numerical results indicate that the proposed algorithm significantly outperforms conventional RIS scheme and the benchmark schemes.