Multilayer Concept Drift Detection Method Based on Model Explainability

Timely detection of concept drift plays a vital role in ensuring the stability and reliability of data-driven models. However, existing concept drift detection methods face challenges in achieving a proper balance between accuracy and timeliness while also disregarding the precise localization of the drift point. To address these issues, this paper presents a novel three-layer drift detection algorithm named Hierarchical Concept Drift Detection based on SHapley Additive exPlanations (HCDD-SHAP). Initially, concept drift is identified by monitoring changes in the error rate, ensuring the promptness of drift detection. Subsequently, the SHAP method is incorporated to assess the features’ contribution to drift, thereby enhancing the precision of drift detection. Lastly, a drift point backtracking mechanism is introduced to accurately pinpoint the initial drift point, facilitating the rapid creation of new models. Experiments show that HCDD-SHAP exhibits superior performance on synthetic and real-world datasets, surpassing conventional drift detectors in terms of precision, recall, and detection delay across diverse detection scenarios. Additionally, in the context of a shield construction project, HCDD-SHAP demonstrates its ability to swiftly identify variations in dynamic engineering conditions, which underscores the considerable advantage of the proposed algorithm in scenarios that require real-time monitoring and control.