A Multi-Agent Deep Reinforcement Learning Based Handover Scheme for Mega-Constellation Under Dynamic Propagation Conditions

With the rapidly increasing number of satellites, the handover scheme design is critically important for the low Earth orbit (LEO) satellite networks, especially for the mega-constellations that include massive number of LEO satellites. However, the existing handover schemes for LEO satellite networks are designed based on the static propagation conditions, which cannot satisfy the dynamic feature of communication environment caused by the mobility of LEO satellites and users. To address this issue, a centralized adaptive intelligent handover scheme for mega-constellations is proposed, where the dynamics of the propagation conditions and limited LEO satellite capacity are taken into considerations. Specifically, we first use a three-state Markov model to characterize the dynamically varying propagation conditions between satellites and users. Then, the Loo model is employed to describe the dynamic land mobile satellite channels. By considering the user transmission rate requirement and the load-balancing demand of satellites, we design the user utility function and formulate an optimization problem that aims to maximize the overall long-term utility of the network. To reduce the handover decision-making complexity, a multi-agent successive hysteretic deep Q-learning algorithm is developed and it can efficiently solve the formulated problem by reducing the state and action space. To reduce the signaling overhead and the computation complexity of the proposed centralized handover scheme brought to the control center, a distributed intelligent handover scheme is further developed, where each user is enabled to independently make the handover decision only based on the local information. Simulation results show that both the proposed centralized and distributed approaches can efficiently improve the network performance over the existing schemes.