Learning Based Cooperative Transmission Strategy for Space-Air-Ground Integrated Networks

Unmanned aerial vehicles (UAVs) have been widely applied as efficient aerial relays for satellite-to-terrestrial transmission so that data packets can either be transmitted directly from satellites to users or relayed via UAVs. To well exploit the advantages of such space-air cooperative transmission, data transmission and resource allocation should be jointly designed to fully utilize the constrained resources in space-air-ground integrated networks (SAGINs). However, this problem is rather challenging due to its mixed integer nature (i.e., discrete transmission strategy and continuous resource allocation), high dynamics and large scale of SAGINs. In this paper, we propose a cooperative transmission strategy (MCTS) based on Multi-agent reinforcement learning, which alleviates the curse of dimensionality in actions through value function factorization. Specifically, we model the joint transmission strategy and resource allocation as a decentralized partially observable Markov decision process with the aim of maximizing the long-term achievable system throughput. For each agent, we first employ a Parameterized deep-Q-network (P-DQN) algorithm to effectively decompose the coupled discrete transmission strategy and continuous resource allocation. To deal with the non-stationary environment, we further adopt a QMIX framework to aggregate the local Q-value of agents. Simulation results demonstrate the scalability and superiority of the proposed MCTS over two benchmark algorithms in terms of achievable system throughput.