A Survey of Multi-Agent Reinforcement Learning for Cooperative Control in Multi-AUV Systems

The increasing demand for adaptive and autonomous smart ocean systems has accelerated the deployment of Autonomous Underwater Vehicles (AUVs) in various complex underwater missions, including collaborative navigation, target tracking, data collection, and energy management. To address the inherent challenges of the underwater environment, such as the limitations of acoustic communication, unpredictable environmental dynamics, and partial observability, Multi-Agent Reinforcement Learning (MARL) with the Centralized Training with Decentralized Execution (CTDE) paradigm has emerged as a promising solution. This paper presents a focused survey on MARL algorithms and their applications in multi-AUV systems. We classify the algorithms into value-based approaches (e.g., VDN and QMIX), policy-based approaches (e.g., Multi-Agent Deep Deterministic Policy Gradient (MADDPG) and Multi-Agent Proximal Policy Optimization (MAPPO)), and hybrid approaches that support explicit communication (e.g., CommNet, DIAL, and ROMA). We further review several relevant simulation environments, such as PettingZoo, UWSim, and Gazebo, along with underwater acoustic channel modeling and key performance metrics used for system evaluation. Complementing the survey study, we present representative experiments in data collection and energy efficiency scenarios using MADDPG and MAPPO algorithms. The results include comparative analyses of average reward, overflow, Flight eXceedance (FX), and energy consumption, accompanied by trajectory visualizations in a grid-based simulation environment. Finally, we provide a synthesis of future research directions, including the integration of Meta-Reinforcement Learning (Meta-RL), graph-based role allocation, and 3D physics-based simulator training, to strengthen the adoption of MARL in autonomous ocean systems. This survey is expected to serve as a reference point for further research in developing AUVs that are smarter, more collaborative, and resilient to extreme conditions.