A Dynamic and Collaborative Spectrum Sharing Strategy Based on Multi-Agent DRL in Satellite-Terrestrial Converged Networks

Satellite-terrestrial spectrum sharing is an effective method to alleviate the scarcity of spectrum resources. However, with the dense deployment of terrestrial systems, the satellite downlink beam covers multiple terrestrial base stations (BSs) and interacts with terrestrial downlinks during spectrum sharing. The spectrum sharing users encounter co-channel interference of inter-system, inter-cell and intra-cell, posing significant challenges for downlink spectrum sharing in converged system. In this paper, we propose a cooperative multichannel spectrum sharing framework where the multiple terrestrial BSs downlinks share multichannel spectrum resources of low earth orbit (LEO) satellite while guaranteeing the rate requirements of all shared users and aiming at improving spectrum efficiency of converged system. We formulate the cooperative satellite-terrestrial spectrum sharing strategy as a global spectrum efficiency maximization problem for the converged system, achieved through multichannel assignment and power allocation optimization for terrestrial shared users. To solve the non-convex optimization problem, we provide a dynamic and collaborative spectrum sharing (DCSS) algorithm based on multi-agent deep reinforcement learning (MADRL), which includes DDQN-based multichannel assignment and DDPG-based power allocation. DCSS transforms the problem to a decentralized partially observable Markov decision processes (Dec-POMDP) and adopts centralized-training-distributed-execution (CTDE) framework to address the challenge of acquiring global information in real-time. Specifically, each terrestrial BS acts as an agent with a globally trained two-layer joint deep neural network model that can adaptively allocate the multichannel and power based on observed system local information. Extensive simulation results demonstrate that DCSS algorithm outperforms the existing typical algorithms.