Networked Multi-Agent Reinforcement Learning in Continuous Spaces

Many real-world tasks on practical control systems involve the learning and decision-making of multiple agents, under limited communications and observations. In this paper, we study the problem of networked multi-agent reinforcement learning (MARL), where multiple agents perform reinforcement learning in a common environment, and are able to exchange information via a possibly time-varying communication network. In particular, we focus on a collaborative MARL setting where each agent has individual reward functions, and the objective of all the agents is to maximize the network-wide averaged long-term return. To this end, we propose a fully decentralized actor-critic algorithm that only relies on neighbor-to-neighbor communications among agents. To promote the use of the algorithm on practical control systems, we focus on the setting with continuous state and action spaces, and adopt the newly proposed expected policy gradient to reduce the variance of the gradient estimate. We provide convergence guarantees for the algorithm when linear function approximation is employed, and corroborate our theoretical results via simulations.

I. Introduction

Reinforcement learning (RL) has been widely advocated in applications of sequential-decision making under uncertainty. One great challenge in applying RL algorithms to practical systems is that usually the systems involve more than one decision-maker, i.e., multiple agents that interact with each other. This multi -agent setting finds broad applications in practical control systems, including the power grid [1], robotics [2], and unmanned vehicles [3]. In this work, we focus on developing RL algorithms for such a setting, i.e., the problem of multi-agent reinforcement learning (MARL).