Reinforcement Learning-based Position Control for a Disc-shaped Underwater Remotely Operated Vehicle

This paper presents the design and development of a deep reinforcement learning-based controller for a disc-shaped underwater remotely operated vehicle (Disc-ROV). The control process is modeled as a Markov decision process (MDP) with unknown state transition probabilities. One critical issue with conventional RL-based control methods is steady-state error. To reduce the final control error, we introduce an integral method to the actor-critic structure. Another issue is the lack of smoothness in the actions, which can be harmful to the actuators. To address this issue, we introduce an additional penalty to the training loss of policy learning, which eliminates high-frequency components in the control signal. We validate the effectiveness of the proposed algorithm in a simulation environment with a position control task. The simulation results demonstrate the accuracy and stability of the trained position control strategy.