Improving Surface Defect Detection for Trains Based on Visual-Language Knowledge Guidance on Tiny Datasets

Efficient and accurate detection of surface defects on trains is crucial for ensuring train safety. However, the insufficient defect samples and their diverse patterns make defect detection in complex environments highly challenging. This paper proposes a novel train surface defect detection model (ViLG) via visual-language knowledge guidance. By leveraging broad semantic knowledge of CLIP, the model compensates for the insufficient defect semantics in tiny datasets and enhances the ability to recognize unseen defects. First, we propose Visual Feature Guidance with CLIP, which enriches and enhances the global representation capabilities of backbone while preserving its self-learning ability for visual representations. This improves semantic understanding of complex scenarios and diverse defects. Second, we propose Defect Query Selector, which selects defect queries based on the semantic relevance between texts and global feature embeddings. This increases attention to potential defects and reduces missed detections. Finally, we propose Semantic Consistency Loss, which semantically aligns defect queries with defect prompts. With additional cross-modal supervision signals, it refines the semantics of defects. For real-world scenarios with normal reference images, we propose ViLG+, which effectively filters false positives using feature similarity. It further verifies that the global embeddings effectively represent the overall structure of visual scenes as well as subtle local features. Compared with other advanced methods on two train surface defect datasets and two public defect datasets, ViLG shows higher precision, recall, and average precision on unseen defects with relatively faster speed, with average improvements of 23.58, 3.23, and 6.05, and has a more balanced false positive rate and false negative rate.