Online Self-Evolving Anomaly Detection for Reliable Cloud Computing

Production cloud computing systems consist of hundreds to thousands of computing and storage nodes. Such a scale, combined with ever-growing system complexity, is causing a key challenge to failure and resource management for dependable cloud computing. Efficient system monitoring and failure detection are crucial for understanding emergent, cloudwide phenomena and intelligently managing cloud resources for system-level dependability assurance and application-level performance assurance. To detect failures, we need to monitor the cloud execution and collect runtime performance data. These data are usually unlabeled at runtime in real-world systems, and thus a prior failure history is not always available. In this paper, we present a self-evolving anomaly detection framework for cloud dependability assurance. Our framework does not require any prior failure history, and it self-evolves by continuously exploring newly verified anomaly records and continuously updating the anomaly detector at runtime without expensive model retraining. A distinct advantage of our framework is that cloud system operators only need to check a small number of detected anomalies (compared with thousands-millions of system/application event records) and their decisions are leveraged to update the detector. Thus, the detector evolves following the upgrade of system hardware, update of software stack, and change of user workloads. Moreover, we design two types of detectors, one for general anomaly detection and the other for type-specific anomaly detection. Leveraging self-evolution and online learning techniques, our detectors can achieve 88.94% of sensitivity and 94.60% of specificity on average, which makes them suitable for real-world deployment.