Design and Evaluation of Scalable Intrusion Detection System Using Machine Learning and Apache Spark

The dependency on internet and network usage is swiftly increasing from the past few years. Where the dependency is more, the security risks increases and as known cyber-attacks on network communication has increased over a decade against organization firms, government firms and even individuals. Currently, shielding private data, research data and maintaining confidentiality is a critical problem. Hence, it is necessary to develop an intrusion detection system to identify various unknown attacks. As the data flow in network is continuous, capturing it results in huge volume of data so, we need big data technologies like Apache Spark to handle those data and give us the information in short period of time. Machine learning is a powerful investigation structure to distinguish odd occasions happened in the system traffic stream. In this work, we developed Apache Spark and Machine Learning based Network Intrusion detection system which is capable of analyzing the huge data from the network traffic data. We evaluated the performance of Naive Bayes, Support Vector Machine (SVM), Decision Tree and Random Forest classification algorithms for network intrusion detection system using Apache Spark platform. In this paper, we conducted our experiments on UNSW-NB 15 dataset, a recent public dataset for network intrusion detection. Experimental results show that Decision Tree classifier performs better in terms of accuracy in classifying packet traffic as normal or attack and we are also able to determine the type of attack category by using training and testing instances of dataset along with all features. Our model split the dataset into testing and training instances randomly and we calculated accuracy parameters.