XPERT: Empowering Incident Management with Query Recommendations via Large Language Models

Large-scale cloud systems play a pivotal role in modern IT infrastructure. However, incidents occurring within these systems can lead to service disruptions and adversely affect user experience. To swiftly resolve such incidents, on-call engineers depend on crafting domain-specific language (DSL) queries to analyze telemetry data. However, writing these queries can be challenging and time-consuming. This paper presents a thorough empirical study on the utilization of queries of KQL, a DSL employed for incident management in a large-scale cloud management system at M
ICROSOFT
. The findings obtained underscore the importance and viability of KQL queries recommendation to enhance incident management. Building upon these valuable insights, we introduce X
PERT
, an end-to-end machine learning framework that automates KQL recommendation process. By leveraging historical incident data and large language models, X
PERT
generates customized KQL queries tailored to new incidents. Furthermore, X
PERT
incorporates a novel performance metric called X
CORE
, enabling a thorough evaluation of query quality from three comprehensive perspectives. We conduct extensive evaluations of X
PERT
, demonstrating its effectiveness in offline settings. Notably, we deploy X
PERT
in the real production environment of a large-scale incident management system in M
ICROSOFT
, validating its efficiency in supporting incident management. To the best of our knowledge, this paper represents the first empirical study of its kind, and X
PERT
stands as a pioneering DSL query recommendation framework designed for incident management.