Distributed Emergent Agreements with Deep Reinforcement Learning

Building autonomous agents that are capable to cooperate with other machines is an essential step towards large scale application of AI systems. Especially systems comprised of multiple self-interested agents with general sum returns can profit from cooperative behavior as cooperation can help to increase the return from all agents simultaneously. A critical aspect that might undermine cooperation is given if agents cannot make credible threats or promises (called commitment problems). Inspired by this idea in this work we augment deep reinforcement learning agents with the capability to build agreements with one another, thereby enabling agents to autonomously learn at which time to cooperate with other agents. This approach, called distributed emergent agreement learning (DEAL), enables agents to commit to specific policies defined by the agreement. We evaluate DEAL with up to 16 agents, represented as Deep Q-Networks or instances of Proximal Policy Optimization in a factory domain and empirically show that agreements increase cooperation by improving both overall and agent individual returns.