Multi-AUV Pursuit-Evasion Game in The Internet of Underwater Things: An Efficient Training Framework via Offline Reinforcement Learning

In this paper, we investigate the pursuit-evasion game of multiple autonomous underwater vehicles (AUVs) in a complex ocean environment. The pursuer AUVs need to optimize their trajectories to avoid obstacles and dangerous vortex regions in the environment in order to pursue the escaper AUV. Both pursuer and escaper can sense each other with limited detection capabilities for further pursuit or escape. As the underwater pursuit-evasion (UPE) game is a high-dimensional NP-hard problem, we innovatively transform it into a finite-horizon Markov game process and propose a decentralized training and decentralized execution efficient training framework based on offline reinforcement learning. During the training process, we propose multi-agent independent soft actor-critic to facilitate policy improvement and generate offline dataset, and propose multi-agent independent decision transformer for model training in the UPE game. Extensive simulations demonstrate the scalability and generalization ability of our proposed training framework, which can achieve excellent performance in UPE games under different conditions and environments with only a few AUVs participating in policy improvement to generate high-quality offline dataset.