RIS-Aided Secure A2G Communications With Coordinated Multi-UAVs: A Hybrid DRL Approach

Unmanned aerial vehicles (UAVs) have been envisioned as essential technology to enhance the service quality of wireless systems, whereas the security issue is unavoidable. In this paper, a reconfigurable intelligent surfaces (RIS)-aided air-to-ground secure communication paradigm is conceived, where the RIS is used to boost the security of confidential signals from UAVs to ground users. However, robust trajectory and beamforming designs are required to fully reap the secure enhancement capabilities of RIS for UAV links under imperfect channel state information (CSI) of eavesdroppers. Therefore, we formulate a robust minimum multicast rate maximization problem for jointly optimizing the UAVs' trajectories, the active and passive beamforming. The problem is also constrained by the maximum flight duration and the secrecy outage probability (SOP). After an approximate transformation of SOP, we provide an online decision-making framework that combines multi-agent reinforcement learning (MARL) methods with conventional optimization algorithms. To overcome the insufficient learning caused by random rewards and uncertain environments, we propose a novel regularized softmax risk-sensitive QMIX (RES-RMIX) algorithm to guide the UAVs' flight. Simulation results demonstrate that: 1) the proposed RES-RMIX algorithm outperforms the state-of-the-art MARL algorithms; 2) the RIS-aided multi-UAVs system attains significant rate gain over the cases of single UAV and no RIS.