Advanced Reinforcement Learning Based Penetration Testing

Penetration testing is a technique that involves the identification and exploitation of security flaws by simulating real world attacks to find security loopholes. Manual pentesting is a tedious and time consuming process. With applications being developed within weeks and the rapid growth of cybercrime, automated pentesting is the need of the hour. This research focuses on training the pentesting agent using three different reinforcement learning algorithms - Proximal Policy Optimization(PPO), Deep Q-Network (DQN) and Ad-vantageous Actor-Critic (A2C) to discover the suitable technique to perform penetration testing. The goal is to create an agent which is capable of exploiting the vulnerabilities in a host system in an effective manner. The training occurs against various vulnerable machines to identify the optimal exploit path with each algorithm and their convergence is compared. The study finds that Advantageous Actor-Critic has the best convergence rate compared to the performance of Deep Q-Network and Proximal Policy Optimization. The time duration and resource consumption is minimal proving that manual penetration testing is old and outdated and automating the process is tedious but worthy and effective.