Resilient Multimodal Industrial Surface Defect Detection With Uncertain Sensors Availability

Multimodal industrial surface defect detection (MISDD) aims to identify and locate defect in industrial products by fusing RGB and 3-D modalities. This article focuses on modality-missing problems caused by uncertain sensors availability in MISDD. In this context, the fusion of multiple modalities encounters several troubles, including learning mode transformation and information vacancy. To this end, we first propose cross-modal prompt learning, which includes: 1) the cross-modal consistency prompt serves the establishment of information consistency of dual visual modalities; 2) the modality-specific prompt is inserted to adapt different input patterns; and 3) the missing-aware prompt is attached to compensate for the information vacancy caused by dynamic modalities-missing. In addition, we propose symmetric contrastive learning, which utilizes text modality as a bridge for fusion of dual vision modalities. Specifically, a paired antithetical text prompt is designed to generate binary text semantics, and triple-modal contrastive pretraining is offered to accomplish multimodal learning. Experiment results show that our proposed method achieves 73.83% I-AUROC and 93.05% P-AUROC with a total missing rate 0.7 for RGB and 3-D modalities (exceeding state-of-the-art methods 3.84% and 5.58%, respectively), and outperforms existing approaches to varying degrees under different missing types and rates.