Deep Reinforcement Learning for Distributed Dynamic Coordinated Beamforming in Massive MIMO Cellular Networks

Massive multiple-input multiple-output (MIMO) is a key enabling technology for next-generation communication systems. In massive MIMO cellular networks, coordinated beamforming (CBF), which jointly designs the beamformers of multiple base stations (BSs), is an efficient method to enhance the network performance. In this paper, we investigate the sum rate maximization problem in a massive MIMO mobile cellular network, where in each cell a multi-antenna BS serves multiple mobile users simultaneously via downlink beamforming. Although existing optimization-based CBF algorithms can provide near-optimal solutions, they require real-time and global channel state information (CSI), in addition to their high computation complexity. Due to the non-negligible delay of practical backhaul networks and the high-complexity optimization process, it is almost impossible to apply them in mobile cellular networks. Noting that the considered problem under the practical constraints can be modeled as a networked distributed partially observable Markov decision process, we propose a deep reinforcement learning-based distributed dynamic coordinated beamforming (DDCBF) scheme, which enables each BS to determine the beamformers with only local CSI and some historical information from other BSs. Besides, the beamformers can be calculated with a considerably lower computational complexity by exploiting neural networks and expert knowledge, i.e., a solution structure observed from the iterative procedure of the centralized optimization algorithms. Moreover, we provide extensive numerical simulations to validate the effectiveness of the proposed DRL-based approach. With lower computational complexity and less required information, the results show that the proposed approach can achieve comparable performance to the centralized iterative optimization algorithms.