Satellite-Terrestrial Coordinated Multi-Satellite Beam Hopping Scheduling Based on Multi-Agent Deep Reinforcement Learning

Non-geostationary orbit (NGSO) constellations enabled by beam hopping (BH) technology are characterized by wide coverage and high spectrum efficiency. However, how to efficiently schedule multi-satellite beam resources to satisfy the heterogeneous and uneven terrestrial traffic demands remains a huge challenge for satellite operators. This paper proposes a satellite-terrestrial coordinated multi-satellite BH scheduling framework, where the complex multi-satellite BH problem is formulated into a long-term and a short-term subproblems. The long-term subproblem is cell-satellite association problem, which is solved by a low-complexity iterative algorithm executed in network operation control center (NOCC) to minimize the traffic load gap among satellites while considering the interference avoidance. The short-term subproblem is multi-satellite traffic-driven BH problem and we propose a multi-agent deep reinforcement learning (MADRL) architecture where each satellite can cooperatively make real-time BH decisions using the well-trained model by QMIX algorithm to adapt to time-varying and heterogeneous traffic. Simulation results demonstrate that the traffic load gap and network delay have been reduced by 70% and 50% respectively compared with non-load-balancing scheme. Besides, the proposed algorithm outperforms other benchmarks in terms of the network throughput under various traffic load cases and the average network delay is kept within 4 ms. Furthermore, the proposed QMIX-BH can be applied to real-time scheduling since the execution time is less than 1 ms.