Multi-Agent Cooperation Decision-Making by Reinforcement Learning with Encirclement Rewards

Multi-agent decision-making is increasingly applied on many situations especially on military applications, but their autonomous decision-making ability needs to be improved. Multi-agent Deep Deterministic Policy Gradient (MADDPG) adopts the method of centralized evaluation and decentralized execution, while updates each agent's network parameters based on the global state information rather than only its own state, which can make the entire agent's policy network update in the direction of the global optimum, rather than the individual optimum. In the process of multi-agent cooperation decision-making, the encirclement rewards is introduced to guide agents to make cooperative actions and alleviate the problem of sparse rewards. Firstly, we define the encirclement. By using Graham's algorithm, we find out the effective encirclement of
$N$
agents. We evaluate the encirclement quality from the area of the encirclement and the difficulty of breaking-through for the adversary, and then design the rewards function based on this. Simulation experiments show that the convergence speed and win rate of MADDPG algorithm based on encirclement rewards is significantly improved, and it also has strong adaptability to various task scenarios.