Multi-Agent Reinforcement Learning for Collision Avoidance and Path Following of Autonomous Surface Vehicles

Collision avoidance and path following are critical tasks for Autonomous Surface Vehicles (ASV s), particularly in dynamic, multi-agent environments. This paper presents a novel approach to tackling these challenges using Multi-agent Reinforcement Learning (MARL) for decentralized control. Specifically, we employ the Multi-agent Proximal Policy Optimization (MAPPO) [1] algorithm to address the problem of collision avoidance with multiple dynamic obstacles while ensuring effective path following for ASVs [2] [3]. Each agent independently makes navigation decisions without relying on communication with other agents, treating them and any moving objects as dynamic obstacles. In addition to the agents, the environment contains static and random dynamic obstacles, allowing agents to interact with different obstacles during training. This exposure helps them learn more effectively by encountering various scenarios. The agents are modelled using Nomoto first-order dynamics, a standard model for ship manoeuvring. For training, the input state to the reinforcement learning (RL) network consists of a grid representing obstacle positions and angles relative to the agent, as well as the agent's own state information-such as heading error (HE), cross-track error (CTE), velocity (v), and rudder angle. The output of the RL model comprises the rudder angle rate and acceleration, which the agent uses to control its path and avoid obstacles in real time. A comprehensive reward function was developed, focusing on both target approach and obstacle avoidance, encouraging agents to minimize collisions while staying on course. A risk assessment model was developed to avoid unnecessary collision avoidance actions, which dynamically determines the switching time between path following and collision avoidance [4]. The system employs shared policy and critic networks, leading to faster training times and improved scalability, making it suitable for large, complex environments with many agents. Simulation results demonstrate that ASV s can successfully navigate environments with multiple dynamic obstacles, maintaining safe distances and following their intended paths. The system was rigorously tested in a custom simulation environment, built using the Ray, Stable Baselines3, Gymnasium, and other supporting Python libraries, offering a scalable and flexible platform for future maritime applications.