Network Parameter Control in Cellular Networks through Graph-Based Multi-Agent Constrained Reinforcement Learning

Cellular networks are growing in complexity at increasing speed and the geographical locations in which they are deployed in are getting denser. Traditional control methods fall short in providing a scalable and dynamic way of adapting the network to new conditions. Distributed multiagent reinforcement learning successfully addresses scalability problems seen in centralized approaches. The question of achieving learning with constraint satisfaction in distributed systems is still left unanswered in the state-of-the-art. In this work, we aim to perform distributed multi-agent constrained reinforcement learning in order to learn a policy online while satisfying imposed constraints. We use a coordination graph to model the interactions between agents and decompose the global value function. A conservative safety critic is trained in parallel to evaluate the safety of proposed actions. Our method allows for separate training of both the critic and the value network independently of each other, and hence offers flexibility in how and when to train the different models. The results are compared to a baseline using no safety critic. Simulations show that the agents succeed in learning a policy that can satisfy the constraints, while still maximizing the objective.