Self-Supervised Learning for Multimodal Fault Diagnosis With Shapley-Value Weighted Transformers

In recent years, there has been extensive exploration of intelligent fault diagnosis (IFD) methodologies. Nevertheless, there has been a paucity of prior research addressing the integration of multisensor and multimodal information of fault features. To address these challenges, a novel multimodal cross-sensor transformer (MCT) incorporating self-supervised learning (SSL) is proposed for fault diagnosis. Specifically, during the training phase, first, a channel-based embedding (CE) method leveraging self-attention mechanisms is proposed to transform data collected from different sensors, each treated as a distinct channel, into variable tokens, significantly boosting the transformer’s ability to generalize across diverse sensor fault diagnosis tasks. Subsequently, a novel self-supervised multimodal learning approach is devised to investigate the correlation and divergences between time-domain and frequency-domain features, thereby augmenting feature recognition capabilities. Finally, an uncertainty-based multimodal loss weighting (UMLW) mechanism is proposed to dynamically adjust the weights of losses according to their uncertainty, thus concentrating the learning process on the most informative features, which enhances diagnostic accuracy and model robustness. In the model deployment phase, Shapley values are utilized to allocate influence among different modalities, ensuring that key modalities dominate the model predictions. Experimental evaluations are carried out on a publicly available bearing dataset and a dataset collected from a chemical factory, demonstrating that the MCT enhances the $F1$ score by 6.43% to 10.05% and the accuracy by 5.23% to 10% in comparison with state-of-the-art methods.