Decentralized Multi-agent Reinforcement Learning with Shared Actions

In this paper, we consider a multi-agent system with N cooperative agents where each agent privately observes its own private type and publicly observes each others' actions. We propose a novel model-free reinforcement learning algorithm to compute the optimal policies for the agents that maximizes their collective reward. This setting belongs to the broad class of decentralized control problems with partial information. We use the common agent approach [1], wherein some fictitious common agent chooses the best policy based on a belief on the current states of the agents. These beliefs are updated individually for each agent from their current belief and action histories without the knowledge of the system dynamics. In this paper, we employ particle filters, called the bootstrap filters, to update the belief of all agents in a distributed manner. We illustrate our results with the help of a smart-grid application, where the users strive to reduce collective cost of power for all the agents in the grid. Finally, we compare the performances for model and model-free implementation of the reinforcement learning (RL) algorithm establishing the effectiveness of particle filter (PF) method.