Training Environments for Reinforcement Learning Cybersecurity Agents

The development and training of Reinforcement Learning (RL) agents that can be applied to solve cybersecurity related challenges is a growing field of research. However, creating environments to train these agents effectively and efficiently is a nontrivial task. As part of the Markov Decision Process (MDP) involved in training an RL agent, the model must take actions on an environment and observe the impact of these actions on the state of the environment. For RL agents training in the field of cybersecurity, the environment being acted upon is often a computer network. Two primary streams of training environment have been developed for such agents: network emulators and network simulators. Emulators employ Virtual Machines (VMs) to provide a high-fidelity environment for the agent's interactions but suffer in efficiency since the actions of the agent are executed in real-time on the emulated network. Simulators employ abstracted models of the network environment that can rapidly compute the effect of an action on the environment. Simulated environments can train RL agents in much less time but introduce a simulation-to-reality gap that may result in an agent trained in a simulated environment that performs poorly in real environments. This paper reviews current emulated and simulated network environments for RL training.