Peer Incentive Reinforcement Learning for Cooperative Multiagent Games

Social learning, especially social incentives, is extremely important for humans to achieve a high level of coordination. Inspired by this, we introduce this concept into cooperative multiagent reinforcement learning (MARL), to implicitly address the credit assignment problem and promote the interagent direct interactions for cooperations among agents in cooperative multiagent games. In this article, we propose a novel intrinsic reward method with peer incentives (IRPI) based on actor–critic policy gradient. This method can enable agents to incentivize each other for their cooperations through using causal influence among them. Specifically, a novel intrinsic reward mechanism is innovatively designed to empower each agent the ability to give positive or negative rewards to other peer agents' actions through considering the causal influence of the other agents on it. The mechanism is realized by a feedforward neural network through utilizing causal influence between the agents. The causal influence of one agent on another is inferred via counterfactual reasoning using the joint action-value function in MARL. The quality of the influence is assessed via counterfactual reasoning using the individual value function in MARL. Simulations are carried out on two popular multiagent game testbeds: Starcraft II Micromanagement and Multiagent Particle Environments. Simulation results demonstrate that the proposed IRPI can enhance cooperations among the agents to achieve better performance compared with a number of state-of-the-art MARL methods in a variety of cooperative multiagent games.

I. Introduction

With the recent development of deep reinforcement learning (RL) [1], we have witnessed many achievements in building intelligent agents to solve complex multiagent problems [2], [3]: AlphaStar achieves top professional-player-level performance in Starcraft II, [4], OpenAI Five defeats the world champion in Dota II [5], and RL is even used to address real-world applications in a simulated physical world, e.g., coordination of autonomous vehicles [6], traffic light control [7], formation control of unmanned aerial or ground vehicles [8], etc.