Reinforcement Learning Meets Network Intrusion Detection: A Transferable and Adaptable Framework for Anomaly Behavior Identification

Anomaly detection plays an essential role in network security and traffic classification. Many studies have focused on anomaly detection to improve network security, including machine learning and deep learning methods. These methods often require numerous samples and must obtain the results by classifying the entire data set, thereby limiting their inflexibility. Although transfer and multitask learning have achieved some results in the model’s transferability, these methods must manually label or reprocess the test set. These problems limit the application of previous methods in network security management. To solve these problems, we propose a transferable and adaptable network intrusion detection system (TA-NIDS) based on deep reinforcement learning. The interaction process between the agent and the environment varies every time. A small-scale data set can be used to produce many interactive processes. Therefore, robustness is guaranteed when there are few samples. Then, a reasonable reward function allows the agent to learn how to first choose outliers without classifying the entire data set. This makes the TA-NIDS more adaptable to the scene when we prioritize apparent outliers. More importantly, the original features are transformed into the state of the environment, so no requirement exists for the feature dimension. Furthermore, the general rather than the specific state of one data set makes the model transferable to other data sets. The experimental results for IDS2017, IDS2018, NSL-KDD, UNSW-NB15 and CIC-IoT2023 show that the proposed framework maintains good accuracy when prioritizing outliers and transferability are prioritized simultaneously.