Dynamic Laser Inter-Satellite Link Scheduling Based on Federated Reinforcement Learning: An Asynchronous Hierarchical Architecture

The scheduling of the laser inter-satellite links (LISLs) can effectively decrease the energy consumption by deactivating idle LISLs and reduce the network latency by decreasing the average hop count through establishing appropriate dynamic LISLs, but is challenging in mega-constellations due to a large amount of satellites involved. In this paper, a federated multi-agent deep reinforcement learning (MADRL) method for LISL scheduling is proposed, where the global scheduling is decomposed into independent decisions for each satellite. To reduce the substantial communication overhead attributable to MADRL, asynchronous hierarchical federated learning with partial and global aggregations is utilized, transmitting the model parameters rather than status information among low Earth orbit (LEO) satellites and geostationary Earth orbit (GEO) satellites. The movement of LEO satellites maintains the consistency of local sample distribution and the indirect association among different GEO satellites. Therefore, the global aggregation with infrequent occurrence is asynchronous with the partial aggregation. Moreover, during the partial aggregation, the utility of each layer in the local model of the LEO satellite is also evaluated to minimize the uncertainty, allowing the adaptive upload to ensure efficiency and fairness within a limited power budget. The simulation results show that the proposed method reduces the average hop by about two hops and decreases the LISLs number by over 25%. Meanwhile, the communication overhead of LEO satellites is 46.6% of that in the centralized approach, with the training performance ensured.