Multi-Satellite Beam Hopping Based on Deep Reinforcement Learning for LEO Satellite Systems

Beam Hopping (BH) technology uses time division to use a small number of beams to perform hopping services in multiple service positions at the same time, making resource allocation more flexible, improving resource utilization, and reducing costs. However, due to the high mobility of Low Earth orbit (LEO) satellites and the non-uniform distribution of services in their coverage areas, the traditional beam hopping method for high-orbit geostationary satellite systems is not applicable to LEO satellite communication systems. This paper proposes a beam hopping method for LEO satellite communication systems based on multi-agent deep reinforcement learning. Specifically, we have developed a satellite cloud center that incorporates a digital twin network of the physical satellite system. Within this digital twin network, the optimization objective is to balance the load across the satellites and minimize the average queue delay. To address this optimization challenge, we have formulated the problem as a Markov Decision Process (MDP) in this paper. The multi-agent reinforcement learning method is used to make beam hopping decisions for each satellite, which are then sent to the physical satellite network for execution through the gateway. Simulation results show that the proposed method MA-BH has a smaller load gap between satellites, higher system throughput, and lower average queue delay compare to other methods.