Online Forecasting Based Anomaly Detection For Monitoring Large Scale Streaming Data

Anomaly detection for time series data is critical for monitoring the status of ever-growing data sources, e.g., health metrics of data servers. An important element of anomaly detection is time series forecasting, which is often the most time-consuming stage in the system. Existing approaches based on batch processing usually take several seconds or more to process one forecasting task, thus not applicable for large-scale real-time applications. In this paper, we present an online-learning based forecasting algorithm to address the computation bottleneck. It readily handles missing observations and has a constant time complexity, independent of the number of past observations. Our experiments show that it achieves similar level of accuracy as existing approaches while only takes a fraction of computational resources. The proposed algorithm can help us easily monitor tens of thousands of data sources simultaneously with only a small hardware cost.