Leveraging Deep Reinforcement Learning for Automating Penetration Testing in Reconnaissance and Exploitation Phase

Penetration testing is one of the most common methods for assessing the security of a system, application, or network. Although there are different support tools with great efficiency in this field, penetration testing is done mostly manually and relies heavily on the experience of the ethical hackers who are doing it, known as pentesters. This paper presents an automated penetration testing approach that leverages deep reinforcement learning (RL) to automate the penetration testing process, including the reconnaissance and exploitation phases. More specifically, the RL agent is trained with the A3C model to gain experience choosing an exact payload to exploit available vulnerabilities. Ad-ditionally, our RL-based pentesting tool has three main functions: information gathering, vulnerability exploitation, and reporting. The performance of this approach is benchmarked against real-world vulnerabilities in our experimental environments. After training with environmental settings, the RL agent can assist pentesters in quickly identifying vulnerabilities in their own servers. The RL-based approach can mitigate the problems of labor costs and hunger data for automating penetration testing in the system by learning how to execute exploits on its own. The more pentesters who use this tool, the more accurate the pentesting results will be. With outstanding results, this method proves that it can accumulate learning results from previous environments to successfully exploit vulnerabilities for the next exploit in another environment on the first try.