Finding Needle in a Haystack: An Algorithm for Real-Time Log Anomaly Detection with Real-Time Learning

Logs represent the language of any modern real-time system and contain the earliest diagnosable symptoms of failures. The system reliability can be significantly improved by implementing real-time log anomaly detection that captures system deviations early, to apply corrective actions. However, challenges like huge volume of logs, system heterogeneity, lack of labeled data for training, dynamic system behavior etc. pose difficulty to implement such real-time anomaly detection engines on a large scale. This paper proposes a novel, computationally efficient, unsupervised, real-time log anomaly detection algorithm that also learns in real-time. Primarily based on frequency spectrum analysis, it also works in offline mode for historical datasets. Besides detecting anomalous logs, it supplies additional information on anomaly type (temporal, lexical, augmented expertise) and an anomaly score. The paper also discusses algorithm’s hyperparameter tuning and empirical strategies to improve the serviceability for real-world datasets. Experiments demonstrate the effectiveness of anomaly detection and computational performance on different industrial datasets.