Multi-USV Cooperative Formation Control via Deep Reinforcement Learning with Deceleration

The cooperative formation control for multiple unmanned surface vehicles (USVs) is studied in this paper. Virtual leaders are introduced to calibrate reference positions of USVs in the formation. A novel algorithm called deep reinforcement learning with deceleration (DRLD) is proposed to solve the problem of multi-USV cooperative formation control. To maintain the stability of the formation, a deceleration mechanism is first proposed. Invalid actions of USVs are avoided by the deceleration mechanism during the formation process. Then, to avoid sparse rewards, an improved reward shaping method is proposed. The artificial potential field (APF) is introduced to construct reward functions in DRLD. Moreover, to set priorities for collision avoidance in different cases, weights are introduced into reward functions. This priority-based scheme makes the collision avoidance of USVs become more reasonable. The proposed DRLD can still provide a good performance even if the multi-USV system executes formation switching. Finally, comparative experiments and ablation studies are conducted to confirm the effectiveness of the DRLD algorithm.