Malicious Encrypted Network Traffic Detection Using Deep Auto-Encoder with a Custom Reconstruction Loss

Current security solutions face significant challenges in dealing with the ever-increasing complexity and sophistication of cyber-attacks. This is particularly true for the solutions that inherently rely on deep packet inspection techniques for malicious traffic detection, as almost all the network traffic is now encrypted, rendering these methods ineffective. Recently, there has been a surge of research focused on adopting behavior-based detection techniques, such as Anomaly-based Network Intrusion Detection Systems, that leverage machine and deep learning approaches for detecting malicious encrypted traffic. In spite of their promising results, these approaches are still far from being widely deployed in real-world scenarios. In this paper, we have devised a fully-supervised auto-encoder architecture with a custom reconstruction loss in order to effectively model both benign and malicious encrypted network traffic, with the aim of identifying malicious instances. The experimental results demonstrate that our proposed approach outperforms state-of-the-art methods on the UNSW-NB15 dataset, while achieving comparable performance on the CSE-CIC-IDS2018 dataset.