MAAD: A Distributed Anomaly Detection Architecture for Microservices Systems

Anomaly detection plays a critical role in microservices systems by enabling system administrators to promptly detect and respond to anomalies. However, existing anomaly detection systems often necessitate the centralization of log and trace data from diverse system components and rely on resource-intensive statistical methods or deep learning models for analysis. This approach impedes real-time anomaly detection and requires a significant demand on computing resources. In this paper, we design a multi-agent-based, distributed anomaly detection architecture called MAAD to address the limitations. MAAD consists of a collection of agents that cooperate together to identify abnormal behaviors in a distributed manner. Each agent is deployed along with a single service and applies lightweight machine learning techniques to perform local anomaly detection based on its own logs, local context, and information extracted from its parent span service.To preserve the graph information in a microservices request, an agent can communicate essential features with each other, taking into account the collective patterns learned from the prior services. We evaluate the effectiveness of MAAD on two microservices datasets, TrainTicket and MicroSS, and show that MAAD achieved high precision (up to 95.8%) and recall (up to 99.6%), outperforming state-of-the-art centralized anomaly detection approaches. Compared to centralized approaches, MAAD reduces the amount of transferred data before anomaly detection by approximately 88%, facilitating real-time anomaly detection. Furthermore, the lightweight nature of MAAD allows for rapid anomaly detection with minimal impact on microservices systems. Compared to DeepLog, MAAD saves approximately 92% detection time without using GPU accelerators.