Joint Spectrum and Power Allocation in Wireless Network: A Two-Stage Multi-Agent Reinforcement Learning Method

This paper investigates the application of multi-agent reinforcement learning (MARL) algorithm to solve the joint spectrum and power allocation problem (JSPAP) in wireless network. The objective of JSPAP is to optimize the subband selection and transmit power levels for links, with the aim of maximizing the sum-rate utility function. To address the JSPAP with discrete subband selection and continuous power allocation, most existing algorithms rely on a centralized optimizer and the instantaneous global channel state information, which can be challenging to implement in large wireless networks with time-varying subbands. To conquer such limitation, a two-stage MARL algorithm is proposed, which comprises a top layer network for selecting subbands across all links and a bottom layer network for determining the transmit power levels for all transmitters. By utilizing the value decomposition technique in the top layer network, the links can cooperatively select transmission subbands, effectively resolving non-stationarity issues in wireless network. Additionally, in the bottom layer network of the proposed two-stage MARL algorithm, each transmitter selects the transmit power level based solely on the local information, thereby effectively reducing computational burden. Empirical experiments demonstrate the effectiveness of the proposed two-stage MARL algorithm by comparison with the state-of-the-art RL algorithms and fractional programming algorithms.