Fully Distributed Multi-Agent RL Framework for QoS Routing

Classical routing heuristics, e.g., Open Shortest Path First, have several significant issues, such as they are not able to generalize or adapt to heterogeneous environments including dynamics of topology, traffic patterns, and Quality of Service (QoS) requirements. To generalize solutions, network operators recently utilized machine learning algorithms at centralized controllers. However, centralized machine learning solutions are not scalable due to many reasons, such as slow data transfer to the central controller in a large network. Distributed multi-agent systems do not require a tedious and complex central controller while reducing data storage and computation burden as tasks are divided and handled at local servers/computers. In this paper, we present a fully distributed multi-agent system named MADQN, addressing the request provisioning problem, i.e., provisioning a request on a dedicated path satisfying latency and bandwidth requirements. MADQN applies a Deep Q-Network reinforcement learning algorithm to train the agents. Although each agent has its data and policy locally, they can still cooperate with other agents to finish a common routing task that maximizes the total reward. We evaluate the effectiveness of the MADQN with a benchmark network that consists of 100 nodes and 432 directed links, and a dynamic set of thousands of requests. The results show that the agents in MADQN can learn to cooperate and provision 99% of the requests, which is about a 9% improvement against the centralized single agent scheme.