Integrating Time Series Anomaly Detection Into DevOps Workflows

Anomaly detection in the monitoring systems of DevOps environments is crucial for ensuring system reliability, preventing downtime, and maintaining the efficiency of continuous integration and continuous deployment pipelines. Artificial Intelligence (AI)-based solutions for automated anomaly detection in DevOps workflows are attracting growing research interest. However, challenges remain regarding the lack of ground truth data from DevOps systems, as well as difficulties in storing, processing, and visualizing the collected data. Furthermore, most of these datasets are unlabeled, making it unclear what constitutes anomalous system behavior, and no generalized approach exists for selecting the most suitable AI algorithms for anomaly detection in such contexts. To address these challenges, this paper publishes a comprehensive time series dataset from 19 different DevOps test systems, comprising 24 performance metrics sampled twice per minute over 30 days. Moreover, to benchmark the dataset, we first label a subset of the dataset based on feedback from the DevOps experts within the industry context. Then six different algorithms are employed on the dataset and their performance in anomaly detection is evaluated using three different Area Under the Curve (AUC) metrics. Additionally, this paper presents a tool for storing, visualizing, monitoring, and integrating anomaly detection algorithms and makes it available to the community. The performance evaluation results aiming to establish a baseline for further research show that AI-based anomaly detection can significantly benefit DevOps workflows but emphasize the need for algorithm selection and parameter tuning tailored to the specific industry context and dataset.