Applying KSE-test and K-means clustering towards scalable unsupervised intrusion detection

We proposed a scalable outlier detection method to identify outliers in large datasets with a goal to create unsupervised intrusion detection. In our work, the strength of Kolmogorov-Smirnov test and K-means clustering algorithm, both with linear time complexity, are combined to create fast outlier detection. While still maintaining high detection rate and low false alarm rate, our method can easily be paralleled for processing a large data set. The result is then applied with a predefined threshold in order to create efficient intrusion detection. We validated our method against the KDD'99 dataset. With appropriate value of threshold and value of K in KSE test, the results showed the detection rate up to 80% with false alarms less than 10%. While scaling linearly, the accuracy of our method is also improved from those of pure KSE-test-based methods.