SPARK-HUNT: A Distributed Framework for Real-Time Threat Detection Using Ensemble Learning on Network Traffic Data

The paper illustrates SPARK-HUNT, a groundbreaking distributed framework for the real-time detection of cybersecurity threats by means of ensemble learning techniques on network traffic data. The system being proposed is built on Apache Spark's distributed computing functionalities to deliver the processing of high-volume network flows in tandem with a multi-classifier ensemble method to further the enhancement of detection accuracy. Our prototype incorporates five machine learning modelsincluding Random Forest, Gradient Boosted Trees, Support Vector Machine, Logistic Regression, and Deep Neural Networks--all of them being combined via a weighted voting mechanism that impacts notably on the overall threat identification. The experimental assessment effecting the use of the UNSW-NB15 dataset provides a demonstration showing the SPARK-HUNT framework presenting 94.7% overall accuracy in terms of incident detection, which is 23.5% more than the value obtained with traditional systems, and at the same time, it has a low false positive rate of 2.3%. The framework manages to process 35,000 events in a second along with a just 1.8 seconds detection latency and the rate of zero-day attacks it successfully identifies is 84.3%. When it was introduced in a simulated enterprise environment which was processing the traffic of 12TB a day, SPARK-HUNT identified 37 threats that had never been detected before and also generated 76% fewer false positive alerts compared to the previous setup. Therefore, SPARK-HUNT acquires the status of being the very effective solution for the cybersecurity issues that need real-time detection of threats to be carried out at scale.