Piper: A Unified Machine Learning Pipeline for Internet-scale Traffic Analysis

Machine learning has been applied to network traffic analysis for a variety of purposes, including botnet detection. To improve the computational efficiency, several architectures have been proposed to consolidate processes common across multiple applications that use the same traffic data. However, when introducing conventional architectures to real-world traffic analysis at Internet scale, the amount of input traffic data and the variety of output features to represent global access patterns become new challenges. To address the challenges, we have developed Piper, a machine learning pipeline, that consolidates diversified machine learning applications in a highly efficient manner. On top of the consolidated architecture, Piper employs two novel techniques: (1) selective sampling to reduce traffic data efficiently while maintaining prediction performance, and (2) a set of enriched features to extract temporal and spatial characteristics in global traffic. For the evaluation, we have been deploying Piper to detect botnets from internet backbone traffic over nine months. The evaluation has confirmed the effectiveness of Piper in terms of computational performance, prediction performance, and lead time to detect botnets.