Scaling and Hardening XLOG: The SQL Azure Hyperscale Log Service

XLOG is the centralized log service of the SQL Azure Hyperscale (Hyperscale) distributed database-as-a-service (DBaaS) within Microsoft Azure. It is responsible for disseminating transaction log to all clients within the distributed database, such as Page Servers and secondary Compute replicas. As the size of the Hyperscale database increases, the number of XLOG clients also increases, thus presenting scalability challenges in the request handling and IO subsystem. This paper describes how to address scalability challenges by applying various techniques. To solve the thread exhaustion problem, a coroutine-based asynchronous log processing framework was implemented, allowing better management of long-polling requests and eliminating thread starvation. The XLOG Pool, a centralized I/O management component, was introduced to reduce redundant I/Os and improve cold start performance by consolidating log reads across clients. A RateLimiter using a token bucket algorithm was introduced to prevent throttling by Azure storage. To ensure data integrity, several layers of validation, including egress and end-to-end checksum validation, were added to detect and prevent data corruption. Performance evaluations showed significant improvements in log processing and read rates with the new asynchronous framework, XLOG Pool and RateLimiter supporting larger databases more efficiently, enabling Hyperscale to support up to 128 TB size databases in production. This paper illustrates how a real-world cloud database service, responsible for hosting mission-critical applications and managing hundreds of petabytes of data, innovates its logging service to enhance scalability, reliability, and cost efficiency.