Stability is Not Downtime: Comprehensive Stability Evaluation for Large-Scale Cloud Servers in Alibaba Cloud

As one of the most critical cloud products, the stability of cloud servers is of paramount importance. Traditionally, stability is evaluated based on downtime, i.e., the duration when cloud servers are unavailable. However, through our extensive engagement in stability-related work, we have discovered that only 27% of stability issues are related to unavailability, which is in fact a subset of stability. Therefore, in this paper, we propose the Comprehensive Damage Indicator (CDI), comprising three distinct sub-metrics: the Unavailability Indicator, Performance Indicator, and Control-plane Indicator. The CDI is able to evaluate the stability of large-scale cloud servers more comprehensively. In the production environment of Alibaba Cloud, the CDI has been implemented on top of Apache Spark. Over the past two years, we have employed the CDI to guide our stability-related works, including architecture comparison, potential problem detection and operation action optimization. The overall results are very favorable. Taking Fiscal Year 2024 as an example, the three sub-metrics have respectively decreased by 40%, 80%, and 35%.