The Effective Methods for Intrusion Detection With Limited Network Attack Data: Multi-Task Learning and Oversampling

Recently, many anomaly intrusion detection algorithms have been developed and applied in network security. These algorithms achieve high detection rate on many classical datasets. However, most of them failed to address two challenges: 1) imbalanced traffic data with limited network attack, 2) multiple data sources that are distributed in different terminals. In detail, those algorithms assume that there are sufficient network traffic data to train their models for intrusion detection. Due to the network attack traffic is always scarce in the real-world network, this assumption is difficult to satisfy in most cases. In this paper, we use Multi-Task Learning (MTL) and oversampling methods to address those challenges of network intrusion detection. Firstly, we use the MTL method to treat each terminal as a single task, and then use relevant information between different terminals to help learn every single task. Meanwhile, we use the oversampling method to overcome the minority problem of attacks. Through a series of experiments on the latest UNSW-NB15 and CICIDS2018 datasets, this paper verifies the effectiveness of MTL and oversampling methods for network intrusion detection with limited network attack data, where they achieve more than 90% detection rate in different experimental settings.