Improving Network Traffic Anomaly Detection through Data Denoising and Unsupervised Learning

Recent advances in technology have allowed people to be more connected, but it also makes them more vulnerable to cyber threats. Hence, advanced threat detection systems have become more important than ever. We have built upon recent work in unsupervised deep learning and further investigated their two-phase anomaly detection process for network traffic data. This paper extends that two-phase pipeline framework, which uses a first model that is faster and prioritizes sensitivity to reduce the incoming data load and then uses the second model that is slower but more accurate, thereby achieving faster and more accurate predictions than either model alone. We implement autoencoder variants in the first phase and isolation forest variants in the second phase, and we present the performance analysis of various examples of the two methods. To further improve the performance of the pipeline, we introduce in preprocessing a denoising process using a separate autoencoder. Our results confirm that the two-phase prediction pipeline, with carefully fine-tuned hyper-parameters, outperforms the individual standalone models: (1) the pipeline predicts more accurately than an autoencoder without significant loss of prediction speed, and (2) the pipeline predicts faster than an isolation forest model without significant loss of accuracy.