Sequential Detection of Image Defects for Patterned Fabrics

Automatic fabric defect detection has been successfully applied to establish the quality quick response system for the automation of textile production. However, the image complexity and diversity of patterned fabrics have effects on the fabric defect detection, which makes it difficult for automatic quality inspection of textiles. In order to solve this problem, a novel method that sequential detection of image defects for patterned fabrics was proposed in this article. Firstly, the fabric image was segmented adaptively based on the periodic distance of the pattern, which can be used for the determination of repetitive pattern block element in fabrics, so that the defect image blocks could be identified according to the minimum principle of structure similarity; Then, the texture features of defect image blocks were extracted to match the constructed feature dictionary of defect-free image block, the defect position could be located by distance measurement and threshold segmentation. Our experiments show that the proposed method has the advantages of low computational complexity, high detection accuracy and strong applicability, by the comparison with existing approaches for defect detection of patterned fabric. Moreover, the new proposed automatic defect location method can solve the problem of deficiency caused by manual defect marking using existing image inpainting algorithms, it can be used for the automatic image inpainting of patterned fabrics without manual marking to improve the robustness of computer vision-based fabric re-engineering.


I. INTRODUCTION
With the continuous improvement of textile quality requirements and the intensification of international trade competition, the quality of textiles has become a major bottleneck in the development of China's textile industry [1]. At present, the quality inspection of textiles is mainly realized by manual inspection. However, manual inspection results are highly subjective and are not sufficient to maintain high-quality standards in high-speed production. It has been pointed out that only 70% of defects can be detected by manual inspection even by trained inspectors [2]. Therefore, it has become a research hotspot for enterprises to develop automatic textile production equipment to detect fabric defects quickly and accurately instead of manual inspection.
The associate editor coordinating the review of this manuscript and approving it for publication was Larbi Boubchir .
Visual inspection method plays an important role in quality inspection. So far, many visual defect detection methods for different cases have been proposed. For example, Luo et al. [3] uses fusion features and support vector machine to detect wood defects; Sun et al. [4] proposed a weld defect detection method based on machine vision; In [5]- [7], some defect detection approaches were proposed for cases such as railroad, display panel and material. Especially, for the defect detection of textile images, Chan et al. [8] et al. carried out fabric defect detection by observing the change of the Fourier spectrum intensity of fabric image. Siew [9] et al. used the numerical characteristics of the second-order grayscale statistics and the statistical characteristics of the first-order grayscale difference to conduct texture analysis on carpet fabric, and then the wear degree of carpet was detected through feature analysis; Zhang [10] et al. used autocorrelation function to detect the repeated units of the fabric structure, and performed statistics VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ or morphological calculation based on these units, and finally the defect detection of plain color woven fabrics without patterns was realized. These methods have been able to achieve high detection rates, even the highest detection rate among them can reach 95%. However, the early pioneered researches on fabric defect detection [11]- [13] has mainly focused on the detection of plain color with simple structure fabrics, but the less researches could be found for the defect detection of patterned fabrics. Patterned fabric could be defined as a fabric that has repetitive pattern or multicolored units in its design. Because of this, it is difficult for the defect detection of patterned fabric images as the detection results are susceptible to the color or pattern interference of these repeated units [14], [15]. The wavelet golden image subtraction method proposed by Ngan et al. [16] is one of the widely used algorithms. Chen [17] improved the method to effectively segment the defect position on the patterned fabric. However, the detection results were affected by the filtering effect of wavelet and template size, which is not ideal for the detection of small defects; Tsang et al. [18] et al. proposed Elo rating method to realize the defect detection of patterned fabric based on fair matching between the partitions on the image. However, this method contains many parameters and many self-set parameters with poor real-time performance, which is not suitable for automatic production applications; Xu et al. [19] introduced the mean hash feature to represent the structure and gray features of the patterned fabric, and realized the defect detection of patterned fabrics, which can effectively represent the structural characteristics of the periodic patterns. However, the above method tends to detect the defects of simple patterned fabrics with small repeated patterned units. Chang [20] et al. adaptively meshed the fabric image based on the periodic distance of the patterns, and proposed a template-based correction method, which can realize the defect detection of periodic structural images. However, it could not detect the edge part that does not meet the template size, which will not be conducive to the integrated detection of patterned fabrics with large repeated patterned units and high complexity. Therefore, an improved local binary feature was proposed for texture analysis, a new surface defect distribution detection method based on this binary feature was attempted to identify the architectonic stone and patterned fabric defects of repeated patterns [21], it provides a new concept for the defect detection.
This article proposed a novel sequential image detection method to identify the defects for patterned fabrics. This method was based on the idea of step-by-step detection and realized the defect location from the defective pattern unit to its specific defect position progressively, which reduces the influence of repeated pattern units on automatic detection, and improves the accuracy of defect detection. The main contributions of this article can be summarized as: (1) The defect image was adaptively segmented based on the periodic distance of the pattern, and the relative position between the defect-free image block and the testing repetitive pattern block in fabrics was corrected by pixel-level image registration, which provided favorable conditions for the subsequent defect location; (2) The cross-points of the testing image block were segmented based on the yarn information of the defect-free image block, and the specific defect position was located by feature mapping and matching, which improved the detection accuracy of defective patterned fabric image.

II. MATERIALS
A database contains enough defective patterned fabric images are important for the validation of the effectiveness of the proposed method. Considering the particularity and scarcity of the defective patterned fabric images, the method of artificial defect images from reference [22] has been referenced in this article. As shown in Fig. 1, three groups of defective sample images and the ground-truth images [23] were artificially generated through some algorithms, including five types of defects, namely, broken end, hole, netting multiple, thin bar and thick bar.

III. METHODS
A novel method that sequential detection of image defects for patterned fabrics was proposed in this article. The first step is to get the defect image blocks by adaptive image segmentation and similarity discrimination. The second step is to locate the defect position by feature mapping and matching. The specific steps were shown in Algorithm 1.

A. TO GET THE DEFECT IMAGE BLOCKS
It is the first step of the proposed method to identify the defect image blocks according to the adaptive image segmentation and similarity discrimination, which provides favorable conditions for the subsequent defect location, mainly including the adaptive image segmentation [16] and the image registration and recognition.

1) ADAPTIVE IMAGE SEGMENTATION
Taking the size of defect-free image block as the sliding window, and the whole fabric sample image was traversed. As shown in Fig. 2 (a), for a 5 × 5 image, 9 repetitive pattern blocks can be obtained after traversing the whole image with a 3×3 sliding window. According to the previous works, it is very important to identify the repetitive patterned blocks regarding defect detection. Reference [25] provides an improved local binary pattern feature based method to measure the correlation of repeated pattern blocks, which can achieve a good result of pattern recognition and segmentation for the rock surface image. However, the patterned fabric has irregularity and periodicity different with rock surface, the structural difference between the testing image blocks is much more obvious. Therefore, the structural similarity [26], as in (1), was selected in this article to identify the difference between image blocks, to reduce the computation cost of algorithms and improve the detection efficiency.
where, µ x , µ y represent the average gray, δ x , δ y are the gray standard deviation, δ xy is the covariance, and C 1 , C 2 are constant. To avoid the case that the denominator is 0, usually, The reference image which is the most similar to the defect-free image block was selected according to the maximum principle of structural similarity, and then the fabric sample to be tested was re-divided into testing image blocks based on the position information of the reference image. The testing repetitive pattern blocks were shown in Fig. 2 (b), including a 3 × 3 image block and 8 edge image blocks.

2) IMAGE REGISTRATION AND RECOGNITION
There is a dislocated relationship between the testing repetitive pattern blocks and the defect-free image block after adaptive image segmentation. In order to improve the detection accuracy, it is necessary to use pixel-level image registration method [27] to correct the relative position between the defect-free image block and the testing image blocks, respectively. Moreover, it also includes the processing of the edge image block, which preserves and realizes the registration and  recognition of the edge part that does not meet the template size.
Since the structural similarity between the testing repetitive pattern blocks and the defect-free image block directly determines whether the testing repetitive pattern block is the   defect image or not. In order to improve the detection efficiency, it is necessary to set the structural similarity threshold as shown in (2) according to the similarity training value between image blocks. Then the defect image blocks can VOLUME 8, 2020  be identified from the testing image blocks based on the minimum principle of structural similarity.
where, I D is the defective image block, I block is the testing image block, I F is the defect-free image block. In practical training, the defect-free image block was directly selected from the original image. Fig. 3(a) showed the result of adaptive segmentation of a defective image, and the ones marked in red were the selected defect-free image block. Taking the defect-free image as the reference image, the position correction based on image registration and the discrimination of structural similarity were carried out for each testing repetitive pattern blocks. The matching result was shown in Fig. 3(b). It was observed that the structural similarity between the testing repetitive pattern blocks and the defect-free image block was concentrated within a certain range (0.75 to 1). Therefore, the defect-free image block and the defect image block could be distinguished by setting the threshold of structural similarity.

B. TO GET THE POSITION OF THE DEFECTS
The defect image block and defect-free image block after position correction realized pixel-level alignment, so only feature matching algorithm is needed to detect the defect position, mainly including feature dictionary construction and feature matching.

1) DICTIONARY CONSTRUCTION
The texture feature has strong anti-noise ability and of certain local sequence repeatability, so it is an effective method to use texture feature to match fabric images with large differences in texture such as thickness and density. The texture analysis method of Gray Level Co-occurrence Matrix (GLCM) [28] is a more classic method, but it has limitations for pixel-level texture images [29]. Local Binary Patterns (LBP) is an operator used to describe the local texture features of an image, which has significant features such as rotation invariance and gray invariance. The LBP operator has been improved as shown in Fig. 4, which is much more effective for image recognition and detection [30]. Fig. 4 (a) shows a basic LBP Operator with a 3 × 3 neighborhood window, whose neighborhood size is limited and its applicability is not strong. Fig. 4 (b) is an improved circular neighborhood LBP Operator.
Any pixel in the image can be defined as the position at the center of the circular neighborhood f (x c , y c ), and its gray value is g c , and the gray value of other p pixels with  equal spacing in the neighborhood is g p . The gray value of the neighborhood pixel is compared with the gray value of the pixel at the center, and the texture feature LBP value of the neighborhood is calculated according to (3).
In order to obtain richer texture structure information, the paper adopts the method of combining LBP and GLCM to extract the texture features of the fabric image. Moreover, Lin [31] pointed out that the LBP 24,3 operator was the most accurate for image defect detection. As shown in Fig. 5, taking an image with a size of 5 × 5 and a gray level of 8 as an example. The local feature information is extracted from the image using the LBP 24,3 operator, and a total of 26-dimensional data includes 25 uniform patterns and 1 non-uniform pattern h LBP = [t 1 , t 2 , . . . , t 26 ] are obtained. At the same time, the gray level co-occurrence matrix is used to extract the features of the image, and the four-dimensional data includes the average of energy, entropy, contrast, and correlation in four directions of 0 • , 45 • , 90 • , and 135 • h GLCM = [t Ene , t Ent , t Con , t Cor ] are obtained. Therefore, the designed texture feature descriptor is h = [t 1 , . . . , t 26 , t Ene , t Ent , t Con , t Cor ], l = 26 + 4 = 30.
To reduce the amount of calculation and improve the matching speed, taking the cross-points as the detection window, the testing image blocks were segmented according to the gray information of the warp and weft yarn. Then the texture features of the image block were extracted with the size of each cross-point as the detection window. After traversing the whole image, the feature dictionary of the defect-free image block as shown in (4) could be obtained, where M and N are the number of warp and weft cross-points of the image.

2) FEATURE MATCHING
The difference, as shown in Fig. 6, is presented by comparing the texture features of the defect image block and the defectfree image block. There are two main methods for evaluating the similarity between two features: Distance and Similarity [32]. Among them, Euclidean distance is the most common distance measure, cosine similarity is the most common similarity measure, and most of the distance measures and similarity measures are derived from the deformation and derivation of the two. Compared with similarity, distance pays more attention to the difference in the distance or length rather than vector. Therefore, based on the absolute difference calculation of feature values, the Euclidean distance, as shown in (5), is more suitable to be selected in this article to carry out feature matching between cross-point image blocks. Fig. 7 was the dictionary matching results of defect-free image block and defect image block. It was observed that the difference degree of the defect-free image block was within a certain level (less than 0.2), while the matching result of the cross-point image blocks in the defect image block has a small amount of large value. Therefore, the defect-free crosspoint image blocks and the defect cross-point image blocks also could be separated by setting the threshold.
To accurately locate the defect position, firstly, the defective cross-point images shown in Fig. 8 (b) were located by setting the distance threshold shown in (6), and then the threshold of adaptive segmentation between the defect and the defect-free cross-point images was calculated by Otsu [33] shown in (7), and finally the defect outlining images shown in Fig. 8 (c) were obtained.

IV. EXPERIMENTS AND RESULTS ANALYSIS A. EXPERIMENTS
In this experiment, the fabric sample image was adaptively segmented based on the periodic distance of the pattern, and then the defect detection and location of the fabric sample image were realized through the step-by-step detection method of feature matching and threshold segmentation for the identified single defect image block. To verify the effectiveness of the proposed method, the classic Waveletpreprocessed golden image subtraction (WGIS) [34] and Visual salient detection method (Itti) [35] methods, and the state-of-the-art Nonnegative Matrix Factorization (NMF) [36] method, were used to compare the results with the proposed method. Figs. 9, 10, and 11, respectively, showed the defect detection results of different methods on three groups of patterned fabrics. To present the results more intuitively, Fig. 12 showed TPR-PPV graphs between the WGIS, Itti, NMF and the proposed methods. TPR estimates how many true defective pixels are detected in the manually marked defective pixels, while PPV estimates how many true defective pixels are in the detected pixels. In TPR-PPV graphs, the point located closer to the top right corner is regarded as an optimized result.
The WGIS, Itti, NMF and proposed method in this article were compared from the two aspects of detection result images and measurement index evaluation. On the one hand, from the detection results in Figs 9,10 and 11, it could be clearly observed that the detection results of the WGIS and Itti methods were greatly influenced by the fabric patterns. In particular, the detection results of the WGIS method were not only influenced by the effect of wavelet filtering, but also related to the selection of template and threshold; The Itti method could clearly show the defect position, but it is easy to misjudge the pattern information as the defects; The NMF method can determine whether the repetitive pattern block contains defects by comparing and analyzing the coefficient features, so as to realize the location of fabric defect image block. However, this method can only roughly locate the repeated pattern block of defects, but can not accurately locate the defect contour; The method proposed in this article could reduce the influence of background pattern information and improve the detection accuracy by detecting the patterned fabric step by step. On the other hand, from TABLES I, II and III, it could be found that the WGIS and Itti methods were superior to the proposed method in this article in terms of individual TPR values, but the results were seriously affected by background patterns; The measured value of NMF shows that TPR was the best and FPR was the worst, which was caused by the roughness of the detection method; However, the detection accuracy of the proposed method was still optimal, and the ACC values were all over 95%; In the TPR-PPV Graphs, it also could be seen more intuitively that the test value of the proposed method is closer to the top right corner, which means that the measurement index of the detection result image of the proposed method is better.

V. CONCLUSION
In this article, a novel method that sequential detection of image defects for patterned fabrics was proposed. It was based on the periodic distance of the pattern and the self-adaptive segmentation to identify the defect image blocks, and then the feature matching and defect localization of the defect image blocks were carried out. Our experimental results show that the proposed method is effective to reduce the influence of background patterns on defect detection and improve the detection accuracy. The method of distribution detection reduces the computational complexity, which is instructive to the study of complex patterns. At the same time, the existing image inpainting methods need to manually mark the defect image position. In the future, defect detection and image inpainting can be combined to enhance the automatic location and inpainting of defect image.