Hardware Telemetry at Scale: A Case Study on SSDs Endurance Monitoring in Datacenters

Hyper-scale data centers typically possess a vast number of interconnected servers to provide a wide variety of compute, storage and AI services. To deliver reliable infrastructure to users, we employ a large-scale telemetry and monitoring system to closely observe the state of our hardware devices across millions of servers. In this paper, we highlight numerous challenges in hardware telemetry and monitoring at scale. An infrastructure fleet of millions of servers consists of millions of solid-state drives (SSDs) which provide useful telemetry data at scale to aid hardware health monitoring. By leveraging Self-Monitoring, Analysis, and Reporting Technology (SMART) metrics, which is an industry standard for SSD health monitoring. We showcase SysInspector, a hardware telemetry and monitoring framework which enables the monitoring of millions of devices in our fleet. We use SSD endurance monitoring as a case study for the effectiveness of the framework. SSD endurance is a quantifiable measure of SSD’s ability to withstand the cumulative effects of repeated read and write operations over its lifespan. We highlight the importance of SSD endurance monitoring in large-scale infrastructure and demonstrate how we utilize the expected lifetime of system as a reference for monitoring and management of SSD endurance rates in our fleet. Our framework leverages anomaly detection algorithms to identify datacenters or service types at risk of exceeding endurance limits, enabling proactive measures to reduce burn rates and extend drive lifespan. The results of this study showcase the benefits of telemetry-driven SSD endurance monitoring in industrial hyper-scale datacenters.