OCR is All you need: Importing Multi-Modality into Image-based Defect Classification System

Automatic optical inspection (AOI) plays a pivotal role in the manufacturing process. It detects anomalies by analyzing image patterns. Despite its importance, the deployment of models for AOI often faces challenges. These include limited sample sizes, variations among dataset, and camera-poses. Besides, AOI often fails to capture a rich mechanism-parameter information from machines or inside images, which typically benefit AOI classification. To address this, we introduce an external modality-guided defect classification framework, rooted in optical character recognition (OCR), to extract features from images as a external modality to enhance performance, termed OANet (Ocr-Aoi-Net). A key aspect of our approach is the feature alignment between differemt modality features, extracted using a single modality-aware model. We further introduce feature refinement and a gating function in our OANet to optimize the combination of these features, enhancing inference and decision-making capabilities. Experimental outcomes show that our methodology considerably imporoved the defect classification model and maintains robustness even in challenging scenarios.