A HyperLogLog-based Solution to Measuring Sliding Window Flow Spread in High-Speed Networks

Measuring flow spread in high-speed networks has numerous practical applications, where flow spread is defined as the number of distinct packets in a flow. Prior solutions focus on the time model of the fixed window, which, as widely acknowledged, suffers from inappropriate window length setting or inaccurate estimation due to the cut-off of the flow that spans multiple consecutive windows. This paper advocates adopting sliding window model, allowing queries at any time on the flow spread during the most recent period of time. The only prior work in sliding window suffers from low accuracy and malfunctions under hyper-compact scenarios. This paper proposes a novel, compact solution, called Sliding Window Virtual HyperLogLog (SWVHLL). SWVHLL extends the HLL sketch, which is designed for measuring single-flow spread in the fixed window, to multiple flow spread estimation in the sliding window. Trace-driven experimental results show that SWVHLL outperforms existing work significantly by reducing the estimation error by up to 60%.