Synth4Seg—Learning Defect Data Synthesis for Defect Segmentation Using Bi-Level Optimization

Defect segmentation is crucial for quality control in advanced manufacturing, yet data scarcity poses challenges for state-of-the-art supervised deep learning. Synthetic defect data generation is a popular approach for mitigating data challenges. However, many current methods simply generate defects following a fixed set of rules, which may not directly relate to downstream task performance. This can lead to suboptimal performance and may even hinder the downstream task. To solve this problem, we leverage a novel bi-level optimization-based synthetic defect data generation framework. We use an online synthetic defect generation module grounded in the commonly-used Cut&Paste framework, and adopt an efficient gradient-based optimization algorithm to solve the bi-level optimization problem. We achieve simultaneous training of the defect segmentation network, and learn various parameters of the data synthesis module by maximizing the validation performance of the trained defect segmentation network. Our experimental results on benchmark datasets under limited data settings show that the proposed bi-level optimization method can be used for learning the most effective locations for pasting synthetic defects thereby improving the segmentation performance by up to 21.3% when compared to pasting defects at random locations. We also demonstrate up to 2.7% performance gain by learning the importance weights for different augmentation-specific defect data sources when compared to giving equal importance to all the data sources. Note to Practitioners—Data limitations have long been a significant challenge for defect detection in the manufacturing industry. Consequently, the generation of synthetic defect samples is a crucial task. In our research, we developed an easy-to-implement, end-to-end, online differentiable Cut&Paste module that can be integrated into any image segmentation-based defect detection framework. This module can generate synthetic defect samples in real-time during the training of deep neural networks. Additionally, to ensure the quality of the generated synthetic defect samples, we introduced a bi-level optimization method that allows for controllable defect generation. This method is guided by validation set performance to enhance task-specific outcomes. Currently, our proposed method can not only learn optimal defect generation strategies but also identify critical areas on product surfaces for generating synthetic defects.