Deep Reinforcement Learning-based Energy Efficiency Optimization for RIS-aided Integrated Satellite-Aerial-Terrestrial Relay Networks

Integrated satellite-aerial-terrestrial relay networks (ISATRNs) have been considered as a promising architecture for next-generation networks, where high altitude platform (HAP) is pivotal in these integrated networks. In this paper, we introduce a novel model for HAP-based ISATRNs with mixed FSO/RF transmission mode, which incorporates unmanned aerial vehicles (UAVs) equipped with reconfigurable intelligent surfaces (RISs) to dynamically reconfigure the propagation environment and fulfill the massive access requirements of ground users. Our aim is to maximize the system ergodic rate by joint optimizing the UAV trajectory, RIS phase shift, and active transmit beamforming matrix under the constraint of UAV energy consumption. To solve this intractable problem, a deep reinforcement learning (DRL)-based energy efficient optimization scheme by utilizing an improved long short-term memory (LSTM)-double deep Q-network (DDQN) framework is proposed. Numerical results demonstrate the superiority of our proposed algorithm over the traditional DDQN algorithm, on single-step exploration average reward values and other evaluation metrics.