Many-Versus-Many UUV Attack-Defense Game in 3D Scenarios Using Hierarchical Multi-Agent Reinforcement Learning

This paper proposes a deep reinforcement learning (DRL)-based method for many-versus-many attack-defense games involving unmanned underwater vehicles (UUVs) in 3D space, focusing on training a defense team to counter attackers. The attackers benefit from speed and unpredictability, while defenders leverage numerical superiority. The scenario includes irregular terrain, and UUVs are limited by low-frequency communication. Firstly, a constrained Apollonius model considering UUV 3D motion characteristics is developed to evaluate the repulsive effect of defenders on attackers. Secondly, a hybrid 3D UUV maneuvering framework integrating end-to-velocity and hierarchical approaches is proposed to reduce the complexity of decision-making strategy learning, enabling UUVs to counter multi-attacker threats and learn repulsion strategies across sub-teams. Thirdly, a scalable learning architecture is designed to adapt to different game scales, with an improved update method to enhance advantage and credit estimation efficiency while ensuring convergence. The combination of population expansion-curriculum training and asynchronous parallel training strengthens the generalization of strategies across various environments. Finally, through comparative analysis with mainstream MADRL-based methods, as well as ablation studies on the framework and rewards, our scheme demonstrates superior learning efficiency and generalization ability. Adversarial experiments across different game scales, along with specialized performance tests, indicate that the defense group exhibits strong robustness and adaptive characteristics.