Pixel-to-Action Policy for Underwater Pipeline Following via Deep Reinforcement Learning

In this paper, we investigate the problems of vision-based pipeline following for a fully actuated autonomous underwater vehicle (AUV) that moves in the horizontal plane. The problems cannot be solved by most of model-based approaches. To this end, we firstly formulate the pipeline following problems for AUVs as continuous-state, continuous-action Markov Decision Processes (MDPs) under unknown transition probabilities. The pipeline following policy is modeled as a mapping from the images produced by the cameras to the velocity of AUVs, and is represented by deep neural network. And then we use Proximal Policy Optimization (PPO) to train the network, obtaining a pixel-to-action policy. Finally, we construct several experiments to verify the effectiveness of the presented approach and the generalization ability of the learned policy. The simulation results show that the learned policy can control the AUV following the pipelines, and have great generalization ability for new and unseen pipeline geometries.