Conditional Variational Auto-Encoder and Extreme Value Theory Aided Two-Stage Learning Approach for Intelligent Fine-Grained Known/Unknown Intrusion Detection

Promptly discovering unknown network attacks is critical for reducing the risk of major loss imposed on organizations and information infrastructure. This paper aims at developing an intelligent intrusion detection system capable of classifying known attacks as well as inferring unknown ones. To achieve this, we formulate the problem of fine-grained known/unknown intrusion detection as a two-stage minimization problem, where the first stage is to seek a score measure for minimizing the empirical risk of misclassifying the known attacks, while the second stage is to find another score measure for minimizing the identification risk of inferring unknown attacks. The hierarchical nature of problem formulation allows us to employ the class conditioned auto-encoders to construct a hierarchical intrusion detection framework. Since the reconstruction errors of unknown attacks are generally higher than that of the known attacks, we further employ extreme value theory in the second stage to model the distribution of reconstruction errors for differentiating known/unknown attack. To further reduce the false positive rate, we add a benign clustering module for learning the multimodal distribution of benign traffic. We conduct an experiment on two widely used datasets for assessing intrusion detection. The results show that the proposed method improves the detection rate of unknown attacks while keeping a low false positive rate.