Decomposed and Prioritized Experience Replay-based MADDPG Algorithm for Multi-UAV Confrontation

Multiple unmanned aerial vehicles (Multi-UAV) confrontation is becoming an important combat mode in the era of intelligent air combat. The confrontation highly relies on the intelligent collaboration and real-time decision-making of the UAVs. Thus, we have proposed a decomposed and prioritized experience replay (PER)-based multi-agent deep deterministic policy gradient (DP-MADDPG) algorithm for the intelligent attacking decision-making of the UAVs in the multi-UAV confrontation scenario. Specifically, the problem of decision-making is formulated as a partially observable Markov Game. To solve the problem, the DP-MADDPG algorithm is proposed by integrating the decomposed and PER mechanisms into the traditional MADDPG. The decomposed mechanism improves the MADDPG framework with local and global dual critic networks to avoid the convergence to a local optimum and a single dominant policy. The PER mechanism optimizes the sampling efficiency from the experience replay buffer for the training process of MADDPG. Simulations have been conducted based on the Multi-agent Combat Arena (MaCA) platform, and the results indicate that due to the decomposed and PER mechanisms, the proposed DP-MADDPG improves the convergence rate and the reward value as compared to the traditional MADDPG and independent learning DDPG (IL-DDPG). Furthermore, the win rate of the DP-MADDPG empowered red party has been improved to 96% in a multi-UAV confrontation scenario.