Online Internet Traffic Monitoring and DDoS Attack Detection Using Big Data Frameworks

Owing to the explosive growth of Internet traffic, network operators must be able to monitor the entire network situation and efficiently manage their network resources. Traditional network analysis methods that usually work on a single machine are no longer suitable for huge traffic data owing to their poor processing ability. To cope with high speed streaming data, various stream-processing-based big data frameworks, such as Storm, Flink, and Spark Streaming, have been proposed. In this paper, we treat network traffic as a streaming data, and propose an online Internet traffic monitoring framework based on Spark Streaming and Flink, respectively. The framework could be used for real-time TCP performance monitoring and DDoS detection. We conduct typical experiments to compare the performance of Spark Streaming and Flink. The experiments show that our framework performs well for large Internet traffic measurement and monitoring.