Intelligent Gateway Selection and User Scheduling in Non-Stationary Air-Ground Networks

With space, air and ground multiple layers, space-air-ground integrated networks (SAGINs) have been emerging as a promising technology to improve coverage and quality of service (QoS) for mobile users. With inhomogeneous access technologies at different layers in SAGINs, the joint gateway selection and user scheduling (GSUS) plays a crucial role to improve QoS and system performance. However, the moving aerial access point leads to highly dynamic inter-layer links, and it is challenging to capture the dynamics when solving the GSUS problem. In this paper, we resort to a non-stationary Markov Decision Process (MDP) formulation to make intelligent GSUS decisions in dynamic SAGINs. Unfortunately, conventional reinforcement learning (RL) is not applicable to solving the non-stationary MDP problem. To this end, we use the Dynamic Parameter Markov Decision Process (DP-MDP) to decompose the non-stationary MDP into a sequence of stationary MDPs, and then encode them with latent parameters, facilitating policy transfer between similar MDPs. Finally, the GSUS problem is solved by using an online learning framework including representation learning and RL. Simulation results demonstrate that the proposed framework outperforms a known benchmark scheme in terms of network throughput and packet drop rate.