EventMon: Real-Time Event-Based Streaming Network Monitoring Data Recovery

Data recovery is a fundamental task for sparse network monitoring with a significant impact on many downstream tasks, such as congestion control, network capacity planning, and traffic engineering. To better capture the network dynamically and quickly respond to network failure, network monitoring systems take finer temporal granularity to collect data to form a real-time view of the network. Unfortunately, the current data recovery for network monitoring relies on matrix and tensor completion algorithms, which fail to satisfy the requirements of real-time recovery. To combat this problem, in this paper, we propose Real-Time Event-based Streaming Network Monitoring Data Recovery (EventMon) that achieves ultra-low latency data recovery in network measurement data streams. Specifically, we leverage a mixture of offline and online architecture, with the offline component learns to capture the historical spatial-temporal correlation while the online component is a novel streaming encoder that updates factor matrices incrementally. To enable training our event-level stream processing module, we devise a novel Stream2Batch algorithm to enable mini-batch style training and ensure the encoder generates the same results with one-by-one stream processing. We conduct extensive experiments on three network monitoring datasets and our evaluation and analysis of the experimental results demonstrate shows that the proposed method outperforms existing schemes in terms of accuracy, inference latency, and high processing throughput.