Green Cell-Free Massive MIMO: An Optimization Embedded Deep Reinforcement Learning Approach

Cell-free massive multiple-input multiple-output (MIMO) deploys a large number of distributed access points (APs) without cell edges, offering seamless connectivity with significantly increased spectral efficiency and system capacity. However, a dedicated fronthaul link is required to connect each AP to a central processing unit (CPU), and the transmissions and hardware-related static power supplies of the APs consume huge energy, resulting in degraded energy efficiency. In this paper, we develop a green energy scheme by simultaneously optimizing power allocation and AP selection. We formulate it as a non-convex mixed-integer nonlinear programming problem (MINLP), which is NP-hard. To address this challenging problem, we propose a new algorithm that embeds non-convex optimization into contemporary deep reinforcement learning (DRL), referred to as optimization-embedded soft actor-critic with graph transformer networks (OSAC-G). OSAC-G enjoys the benefits of directly online inferring solutions for the non-convex problem with a much lower computational complexity compared to conventional non-convex optimization. To enable OSAC-G to adapt to the moderate variations of Class II parameters without learning from scratch, we further develop a graph transformer network (GTN) that extracts the underlying relationships between APs and UEs from a constructed heterogeneous graph. Simulation results demonstrate that the green energy scheme significantly decreases energy consumption compared to the existing ones.