Distributed intrusion detection system in the cloud environment based on Apache Kafka and Apache Spark

After, the emergence of cloud computing (CC), it’s gained more attraction to be used for organizations and users. CC allows to migrate the computing power to the internet services. That makes cloud system target of attackers to disrupt services or data breaching. Many existing works try to deal with security issues in cloud computing systems, but it is still suffering against new updated attacks. Therefore, it’s necessary to develop new IDS able to detect attacks with high performance. In this paper, we present a distributed IDS based on big data tools and machine learing algorithms to detect attacks in the cloud systems. This proposed system designed to be installed in the front of cloud network architecture. The network traffic is collected from edge routers and streamed with Kafka component to Spark component for prepressing, anomaly detection and attack classification. In preprocessing stage, data cleaning, formatting and feature selection based on K-means clustering are performed. In the anomaly detection and attack classification, we compared different machine learning algorithms optimized with hypermeters tuning based on grid search. Various experiments are conducted on Google cloud platform to evaluated the system using CIDDS-001 dataset. The Decision tree classifier outperform all in term of accuracy and F1-score in anomaly detection stage the same for attacks classification stage, Random Forest yielded Decision tree in term of accuracy and F1-score. Duo to lower detection time, we choose DT to build our system.