Learning Adaptive Jamming and Beamforming for Hybrid IRS-assisted Secure NOMA Transmissions

In this paper, we investigate hybrid passive and active intelligent reflecting surface (IRS)-assisted secure non-orthogonal multiple access (NOMA) networks. Multiple users concurrently transmit sensitive data to an access point (AP) in the presence of an eavesdropper (Eve). The hybrid IRS is employed to enhance the NOMA users’ sum rates while simultaneously performing jamming beamforming against the Eve by optimizing the communication channels of NOMA users and injecting controllable noise into the Eve’s channel. We formulate a sum secrecy rate maximization problem by jointly optimizing the users’ scheduling policy, the hybrid IRS’s working mode and beamforming, and the AP’s receiving beamforming. To address combinatorial user scheduling and high-dimensional beamforming design, we develop a dual-cycling deep reinforcement learning (DRL) framework. We first determine the NOMA users’ scheduling strategy and the hybrid IRS’s working mode using a proximal policy optimization (PPO)-based learning algorithm. Then, we optimize the AP’s receiving beamforming and hybrid IRS’s beamforming strategies using an alternating optimization (AO) algorithm. The joint beamforming optimization can significantly enhance the DRL’s learning efficiency by limiting its action space. Moreover, we propose a lightweight two-phase algorithm with approximation techniques to reduce computational complexity by eliminating double-nested loops in AO, while maintaining secrecy performance close to optimum. Numerical results demonstrate that the proposed dual-cycling DRL scheme achieves 54.85% gains in the secrecy rate compared to traditional DRL schemes.