Observer-Based Multi-Agent Reinforcement Learning for Pursuit-Evasion Game With Multiple Unknown Uncertainties

This paper aims to investigate the challenging problem of a multi-agent game with multiple pursuers and a single evader in an environment with multiple unknown uncertainties. A coupled approach combining decentralized observers and reinforcement learning (RL) controllers is proposed to deal with this scenario. Firstly, decentralized observers driven by auxiliary control laws are introduced to estimate the states of uncertain systems, with their best responses obtained through the adaptive dynamic programming (ADP) method. The estimated states, which reflect the actual states of the pursuers’ systems, are concurrently transmitted to the RL controllers. Subsequently, the controllers are trained with observer-based heterogeneous-agent proximal policy optimization (OHAPPO) algorithm, in which a novel global multi-function cost is designed. The algorithm utilizes the advantage decomposition for policy updates in the way of credit assignment, resulting in more stable and efficient updates compared to traditional value decomposition. Moreover, to further enhance the performance of both observers and controllers, a sequential game is established between them, where observers’ policies are influenced by controllers’ optimal control and vice versa. Finally, the simulation results verify the effectiveness of the designed OHAPPO algorithm in the pursuit-evasion game. Note to Practitioners—Numerous military applications, such as missile interception and pursuit or encirclement on sea and land, can be effectively delegated to a group of experienced and intelligent agents. These agents offer safety and efficiency advantages in complex and dynamic adversarial environments. RL methods are good at dealing with such scenarios, but their application in complex real-world scenarios is often hindered, as multiple unknown uncertainties in the environment can affect the effectiveness and stability of strategies, as well as the generalization of simulation to reality transfer and the high cost of training directly in real environments. This study takes the pursuit-evasion game as an example task, forming a universal coupled control method. This method allows active observers with game properties to overcome multiple unknown uncertainties. It enables RL to be trained in relatively real environments, overcoming application limitations caused by uncertainty and facilitating RL research in real scenarios. Due to hardware limitations, the next step has not yet been taken, and it will address testing and deployment issues in real-world environments.