ATMoS: Autonomous Threat Mitigation in SDN using Reinforcement Learning

Machine Learning has revolutionized many fields of computer science. Reinforcement Learning (RL), in particular, stands out as a solution to sequential decision making problems. With the growing complexity of computer networks in the face of new emerging technologies, such as the Internet of Things and the growing complexity of threat vectors, there is a dire need for autonomous network systems. RL is a viable solution for achieving this autonomy. Software-defined Networking (SDN) provides a global network view and programmability of network behaviour, which can be employed for security management. Previous works in RL-based threat mitigation have mostly focused on very specific problems, mostly non-sequential, with ad-hoc solutions. In this paper, we propose ATMoS, a general framework designed to facilitate the rapid design of RL applications for network security management using SDN. We evaluate our framework for implementing RL applications for threat mitigation, by showcasing the use of ATMoS with a Neural Fitted Q-learning agent to mitigate an Advanced Persistent Threat. We present the RL model’s convergence results showing the feasibility of our solution for active threat mitigation.