Hierarchical Reinforcement Learning for Submarine Torpedo Countermeasures and Evasive Manoeuvres

Modern naval warfare environment is becoming increasingly complex, with acoustic-based torpedoes being the most significant threat to submarines. It is essential to develop advanced technologies to enhance submarine survival rates. In this paper, we propose a hierarchical multi-agent reinforcement learning scheme and a realistic underwater simulation environment for optimal submarine torpedo countermeasures and evasive manoeuvres. Our hierarchical model consists of high-level and low-level agents. The high-level agent decides on decoy launches, while the low-level agent executes specific torpedo countermeasures and evasive manoeuvres. We implement underwater simulation environment based on a 6-DOF motion model to realistically simulate underwater object movements and use PID control for accurate and stable physics. This database is used for active and passive SONAR detection of torpedoes and submarines, enhancing the realism of the acoustic environment. We designed 4-level metrics to systematically analyze model performance in static and dynamic environments with single and multiple torpedo scenarios. Also, we propose a new training methodology to address delayed and sparse reward problems by considering submarine manoeuvring characteristics. Experimental results show that our proposed hierarchical architecture demonstrates competitive performance, achieving a survival rate of 89.07% even in the most complex dynamic environment. We demonstrate improved submarine torpedo countermeasures and evasive manoeuvre performance through stable training in complex underwater environments.