Online Solution for Orbital Pursuit-evasion Game via Heterogeneous Proximal Policy Optimization

This paper proposes an online algorithm for multi-player orbital pursuit-evasion games. First, the problem formulates the optimal strategies for both sides among numerous players. Therefore, Nash equilibrium is established. Then, optimal commands for each player can be derived from the saddle-point trajectory. However, traditional algorithms for the corresponding two-point boundary value problem become computationally inefficient as player numbers increase. Hence, multi-agent proximal policy optimization policies are employed to map states to optimal strategies for real-time missions. The Markov game model comprises the states of all players and actions of respective costate variables. The reward based on the extended boundary condition ensures optimality and constraint adherence. Besides, a sequential update is devised due to the heterogeneity of pursuit and evasion policies. Pursuers alter their policies based on current evasion strategies, and evaders adjust accordingly. This update contradicts traditional independent updating and facilitates the convergence of training. Numerical simulations demonstrate that the proposed method exceeds shooting methods in the optimality index and surpasses the collocation in computing efficiency, with the advantage growing as the player count rises. Furthermore, the method exhibits robustness against opponents employing suboptimal strategies and deviation in states by feedback implementation. The results highlight the method's applicability to practical scenarios.