Towards a Standard Benchmarking Framework for Domain Adaptation in Intelligent Fault Diagnosis

Domain shift is a major problem facing contemporary data-based intelligent fault diagnosis (IFD) solutions. While domain adaptation (DA) methods have been proposed to address this issue, standardizing DA benchmarks has not received much attention. Existing studies often use ad-hoc evaluation methods with inconsistent dataset partitioning, labeling, and evaluation criteria. We propose an integrated benchmarking framework to bridge the gap between DA development and evaluation research. Our framework incorporates domain shift key factors such as operating conditions (OCs) and fault levels (FLs) into dataset partitioning, labeling, and evaluation phases. The fault dataset is split into distinct subsets using FLs and OCs as partitioning parameters, while maintaining the original diagnostic classification labels. The DA method is comprehensively evaluated under controlled conditions using permuted subset pairs of the partitioned dataset. The framework is applied to two popular datasets – the Case Western Reserve University (CWRU) and Paderborn University (PU) bearing fault datasets. We demonstrate the framework’s capabilities and application mechanisms using a CNN classifier and an adversarial DA algorithm. Benchmarking results reveal significant accuracy drops from 99% to 36% and 45% with variances of 15% and 20% for the CWRU and PU datasets, respectively, under controlled domain shift conditions. The proposed framework establishes a foundation for standardizing IFD DA benchmarks, bridging the gap between development and evaluation research.