Performance analysis of network intrusion detection schemes using Apache Spark

Fast and efficient network intrusion detection is a very challenging issue as the size of network traffic has become increasingly big and complex. A real time intrusion detection system should be able to process large size of network traffic data as quickly as possible in order to prevent intrusion in the communication system as early as possible. In this paper, we have employed five machine learning algorithms such as Logistic regression, Support vector machines, Random forest, Gradient Boosted Decision trees & Naive Bayes for detecting the attack traffic. For processing and detecting the attack traffic as fast as possible, we have used Apache Spark, a big data processing tool for detecting and analysis of intrusion in the communication network traffic. Performance comparison of intrusion detection schemes are evaluated in terms of training time, prediction time, accuracy, sensitivity and specificity on a real time KDD'99 data set.