Stream-Based IP Flow Analysis

As the complexity of Internet services, transmission speed, and data volume increases, current IP flow monitoring and analysis approaches cease to be sufficient, especially within high-speed and large-scale networks. Although IP flows consist only of selected network traffic features, their processing faces high computational demands, analysis delays, and large storage requirements. To address these challenges, we propose to improve the IP flow monitoring workflow by stream-based collection and analysis of IP flows utilizing a distributed data stream processing. This approach requires changing the paradigm of IP flow data monitoring and analysis, which is the main goal of our research. We analyze distributed stream processing systems, for which we design a novel performance benchmark to determine their suitability for stream-based processing of IP flow data. We define a stream-based workflow of IP flow collection and analysis based on the benchmark results, which we also implement as a publicly available and open-source framework Stream4Flow. Furthermore, we propose new analytical methods that leverage the stream-based IP flow data processing approach and extend network monitoring and threat detection capabilities.