Progressive Prioritized Experience Replay for Multi-Agent Reinforcement Learning

Due to the limitations of load, perception ability and communication range, single agent is difficult to meet the increasingly complex task requirements. As a result, the multi-agent reinforcement learning algorithm may attract more attention. However, the algorithm convergence becomes more difficult with the increase of the agent numbers. In this article, an efficient training framework called Progressive Prioritized Experience Replay (PPER) is proposed to resolve this problem. PPER decomposes the task scene into several similar sub-scenes with a complex degree from easy to difficult. The progressive training (PT) approach is adopted to let the agent accumulate learning experience in sub-scenes before access to the task scene, which greatly reduces the training difficulty. To verify the effectiveness of our training framework, we extended OpenAI gym to create a multi-USV confrontation environment, and the superior performance of PPER has been demonstrated in comparative tests.