Deep learning-based defect detection of metal parts: evaluating current methods in complex conditions

Anomaly detection (AD) plays a key role in automated quality analysis in industrial production. Recent AD methods have shown great potential for the detection of visual defects in several real-world applications. Most of the datatsets used in AD research (e.g. MVTec AD) are composed mainly of images from the laboratory environment with a monochromatic background. Each image contains only one object, which is centred, and its distance and spatial orientation to the camera do not change significantly. However, these conditions cannot be achieved in many realworld manufacturing processes and production lines. In order to test the performance of state-of-the-art (SOTA) AD methods under conditions of variable spatial orientation, position and distance of multiple objects concerning the camera at different light intensities and with a non-homogeneous background, it is necessary to create a new dataset. In this paper, we introduce a new dataset focused specifically on the issue of defect detection during painted metal parts fabrication. Next, we evaluate the performance of current SOTA AD methods on the proposed dataset. Our results show that some SOTA AD methods, which perform well on the standard industrial anomaly detection datatset – MVTec AD, show significantly different performance on our dataset. AUROC image-level difference is up to 23.12%. If we average the scores for all methods on each dataset, we observe the difference of 15.24%. Our experiment shows that for further development and improvement of AD methods, it is necessary to test these methods on datasets based on specific real-world applications.