A Collaborative Anomaly Localization Method Based on Multi-Modal Images

In the context of industrial anomaly detection, anomaly point detection is a challenging task due to the rarity and unpredictable nature of anomalous samples. Existing 2D image-based defect detection methods have certain advantages in capturing features such as texture, color, and shape of parts. However, traditional single-modal defect detection methods (such as using only 2D images or only 3D point cloud data) may have limitations in accurately locating abnormal points when faced with complex surface defects on parts. Therefore, a collaborative abnormal localization method (CALM) based on multi-modal images is proposed to improve the accuracy of anomaly localization by fully utilizing information from multiple data sources. First, we propose a synchronized data augmentation method for 2D and 3D images to address the issue of scarce anomalous samples. Then, feature extraction is performed separately on RGB images and 3D point clouds, leveraging the features from both 2D and 3D images and performing multi-modal feature fusion while aligning the features. Finally, anomaly point localization and segmentation are achieved based on the abnormality scores output by the decoder. To validate the effectiveness of our method, experiments are conducted on the MVTec-3D AD dataset. The Pix-AUROC and Pix-AUPRO means of the CALM method reach 0.909 and 0.739, respectively. The experimental results demonstrate that our method achieves high detection accuracy at the pixel level, outperforming some traditional anomaly localization methods.