A Method for Identifying Flaws on Product Surface Based on Spatial Connectivity Domain

In this paper, we exploit a method for identifying flaws on product surface based on spatial connectivity domain. A number of algorithms for detecting local features exist that were established to enhance the efficiency and accuracy of identifying interest features, such as AKAZE, BFSIFT, BRIEF, BRISK, ORB, SURF, SIFT and PCA-SIFT algorithm. But the data of flaws on product surface which is similar and consistent with the background intensity became a dilemma to detect the feature of image. In terms of identifying flaws on product surface, the above algorithms are not effective and accurate. Our aim is to enhance the accuracy of detecting the feature of flaws on product surface, so that the product with flaws could be accurately identified in industrial production. Therefore, we propose a method to identify flaws on product surface based on spatial connectivity domain. Compared with some other algorithms, such as the extracting texture algorithm, the detecting local feature algorithm and the identifying edge algorithm, our proposed method is more effective and accurate in detecting the local feature flaws on product surface of auto parts in automotive manufacturing factory.


I. INTRODUCTION
We present our latest results as a contribution to detect the local feature of product surface of auto parts in automotive manufacturing. We explore a method for identifying the feature of flaws on product surface based on spatial connectivity domain. The experiment result indicates that the proposed method is more effective than some other methods, such as the accelerated KAZE(AKAZE), binary robust invariant scalable key points (BRISK) and scale-invariant feature transform (SIFT), in distinguishing features of image of product surface for auto parts inspection.
There are currently a number of algorithms for identifying features of the image, such as AKAZE algorithm, BRISK algorithm, oriented features from accelerated seg-The associate editor coordinating the review of this manuscript and approving it for publication was Chao Tong . ment test (FAST) algorithm, rotated binary robust independent elementary feature (BRIEF) algorithm, oriented FAST and rotated BRIEF(ORB) algorithm, speeded-up robust features(SURF) algorithm and SIFT algorithm [1]. In 2004, Lowe and Kong [2] proposed the SIFT algorithm. Although images were rotated, reflected and transformed, the feature of images could be detected via SIFT algorithm. However, the 128-vector descriptors need to be compared, which are complexly computed and matched. An algorithm named BRIEF [3] adopts the FAST keypoint detector and binary robust independent feature descriptor, which have yielded better results in real-time applications. Afterwards, an algorithm named ORB [4] utilizes the FAST and BRIEF algorithm to detect features of image. It performs well even with noise, rotation or affine transform. In 2011 a extracting feature algorithm named BRISK [5] was proposed at the International Conference on Computer Vision (ICCV 2011). It is also a binary feature description algorithm with characters of rotation-invariant, scale-invariant, better robustness, etc. In image registration, BRISK algorithm is faster than SIFT algorithm, and it is also suitable for matching the large fuzzy image. In 2012 a more stable feature detection algorithm named KAZE was proposed at the European Conference on Computer Vision (ECCV). The algorithm constructs a stable nonlinear spatial scale [6]. It solved the problem of boundary blur and detail loss in image processing. KAZE algorithm [7] is a novel feature detection and description method which is invariant to rotation, scale, limited affine transform. One year later, the accelerated version of KAZE algorithm named AKAZE [8] was proposed. It adopts sparse feature vector as key point feature description vector and fastens image processing, so as to enhance the speed of feature extraction and registration.
Some computer experts have also carried out some practical project, and researched how to apply the feature detection algorithm. For example, in the medical breast checking, the SIFT algorithm was used to detect micro-calcification points of breast X-ray film, which accelerates the speed of manual recognition. In medical treatment, a new 3D-SIFT-Flow method was proposed, which takes advantage of dense correspondence using the informative and robust feature [9]. It is effective and efficient for improving the registration accuracy of large tissue deformation and blurry boundary in CT liver image. Mazur et al. [10] applied and optimized SIFT-based dense pixel tracking on 0.35 T cine-MR images during image-guided radiation therapy. Tracking results showed that this method is a high fidelity model in assessing intra-fractional motion on cine-MR acquisitions. In term of detection and description features of image, the core operation of algorithms is to filter the original image to create scale-space [11] images.
In the industry production process, it is necessary to inspect products to make sure that the products come up the standards of quality. For example, in an auto parts factory, Figure 1 is a part of screenshot of product, which was produced by the machine, as shown in Figure 2. In Figure 1(a), it is a qualifying product with flawless, while in Figure 1(b) that is a defective product. Usually, workers detect the product surface by naked eyes whether the product is smooth or not. In Figure 1(b), it is a defective product with two protruding blobs on the product surface, which are signed with blue circle. Although there are some small protruding blobs [12] on the product surface, whose smoothness come up the standards of quality, they are not flaws. Because it took us much time and the labor force to inspect product surface by the artificial processing. This is an encountered problem for inspecting product of auto parts in automotive manufacturing.
The paper is organized into the following sections. In Sect. II, the contribution of the paper is presented. In Sect. III, the problem is analyzed and tentatively solved by several methods. In Sect. IV, the theoretical basis of the proposed algorithm is presented. In Sect. V, description and steps of the proposed algorithm are defined. Finally, the  discussion and conclusions are presented in Sect. VI and VII respectively.

II. CONTRIBUTION OF THE PAPER
According to analysis and experiments of extracting features of the image of product, we conclude that each flaw on the produce surface is composed of a certain number of pixels, and these pixels are connected to form a large area with similar gray value. In this paper, we exploit a method for identifying flaws on product surface based on spatial connectivity domain. In the light of characteristics of flaws on the product surface, we construct Gaussian scale-space [13] and detecting spot feature filter to process the image. Then, according to the distance and weight of adjacent spots, the large spot was stitched by spatial connectivity domain. Finally, in according to the requirements of product inspection, the spot of the flaw is identified, and the value of the flaw which exceed the threshold value (100) is outputted. On the GitHub website, the code of our proposed algorithm will be available at: https://github.com/zhangquanyou/Identifying-Flawson-Product-Surface.git

III. ANALYSIS OF PROBLEM
There are salient object detection(SOD) and general object detection(GOD) in image object detection [14]. But in Figure 1(b), it is difficult to identify the flaw object which signed blue circle label. Because the texture of background of the image is very similar, and the intensity of the feature region is consistent with the surrounding area. So identifying the flaw VOLUME 9, 2021 object is much more challenging than the traditional SOD and GOD. Whether or not to identify flaws by extracting texture algorithms or detecting local feature algorithms or identifying edge algorithms?

A. SHORTCOMING OF EXTRACTING TEXTURE ALGORITHMS
In the aspect of extracting texture feature of image, there are Tamura texture analysis method, Gabor wavelet texture feature extraction method, local binary pattern (LBP) texture feature statistical method and so on [15]. In these methods, it is effective to analyze the surface features of objects by the spatial dependence of gray level. The Tamura texture [16] analysis method is effective to analyze the contrast, coarseness and directionality of the image, but it is poor in recognition of the linelikeness, regularity and roughness of the image. The algorithm of extracting texture feature based on Gabor [17] wavelet transform has better retrieval effect in frequency domain, and it can eliminate redundant information. But the higher the dimension of feature vector increases, the more slowly the retrieval speed becomes. The LBP [18] texture feature statistical method is a kind of analysis method with non-change of the gray level and rotation. It breaks through the difficulty of extracting largescale texture features. Texture of object is the unique feature of object surface, which does not change after rotation or translation or affine transform or other transforms. During our experiment, in order to verify whether Gabor method is valid or not, we construct a Gabor filter to extract the texture feature of flaws on product surface. Figure 3 shows the effect of extracting texture by the Gabor algorithm. As shown in Figure 3, the original image on the left is processed by Gabor filter to get the Gabor image on the right. According to the Gabor image, the similarity of product texture is great, and flaws on the product surface could not be identified by texture algorithms.

B. SHORTCOMING OF DETECTING LOCAL FEATURE ALGORITHMS
According to Figure 1, flaws on the product surface belong to the local features of image. At present, the main popular algorithms of detecting local image features include AKAZE algorithm, KAZE algorithm, BRIEF algorithm, ORB algorithm, SURF algorithm, BRISK algorithm and SIFT algorithm [19] and so on. Each algorithm has its own advantages and disadvantages. AKAZE [20] and KAZE [6] are nonlinear algorithms [32], which cost much time to process image and are not suitable for auto mobile industry production line. BRISK or SIFT or SURF or ORB algorithm [21] is a linear algorithm with fast processing speed. But according to the analysis of detected image features, it is easy to recognize the flawless parts as features. In our many experiments, the local feature algorithm was adopted in processing of identifying flaws on produce surface, resulting in the increasing probability of mistaken discrimination. The parameters of SIFT algorithm were set as below: threshold=0.4, edgeThreshold=10, sigma=1.6. In AKAZE and KAZE algorithm, the parameters were set as below: threshold=0.001, nOctaveLayers=4, nOctaves = 4. The parameters of BRISK algorithm was set as below: nOctaves=4, 9.75<threshold<13.67=10. The experiment results are shown in Figure 4.
According to the experiment results, we found that nondefective points were easily recognized as features of flaws on produce surface. At the same time, considering the automation process of industry production line, it is necessary to quickly inspect the local feature of flaws on product surface. We have absorbed the advantages of linear and nonlinear algorithms and detected features of flaws on product surface by the Gaussian scale-space and Difference of Gaussian (DoG) [22]. The features of flaws more salient than that of other non-defective domains, so that flaws on the product surface can be identified as a blob.

C. SHORTCOMING OF IDENTIFYING EDGE ALGORITHMS
In the aspect of identifying edge of object, there are Sobel operator, Scharr operator, Roberts operator and so on. The Roberts operator, which is known as cross differential algorithm, is used to detect edge lines by local difference calculation. When the edge of object is close to positive 45 degrees or negative 45 degrees, the algorithm is more effective. The disadvantage of algorithm is that the position of edge is not accurate, and the extracted edge lines are thicker. The Sobel operator [23] is often used in noisy images and gray gradient images. According to the gray weighting between the upper and lower pixels and the left and right pixels, the Sobel operator detects the edge of object by the extreme value at the edge. The noisy image could be made smooth and the orientation information of edge is accurately preserved. The effect of Sobel operator's Edge localization is good, but it is prone to widen the edge with multi-pixels, and poor to extract the weak edge. Therefore, in order to effectively extract weak edges, the gap between pixel values must be increased. Scharr operator [24], which is an enhanced Sobel operator, increases the difference between pixel values of edge. Because the inside weighting of filter template is greater than the center of it, the accuracy of edge is higher than that of Sobel operator. Although the principle and usage of detecting edges are the same, the effect of weak edge is different. The experiment results are shown in Figure 5. In addition, there are Prewitt operators, Laplacian operators and so on. The Prewitt operator [25] is very effective in extracting edge of gray gradient image, but it ignores the influence of adjacent points on the current pixel points, which is similar to the Sobel operator. Laplacian operator [26] is very sensitive to noise, so that it is easy to lose the orientation information of some edges. The anti-noise ability of Laplacian is poor, resulting in some discontinuous extracting edges.

IV. THEORETICAL BASIS A. DIFFERENCE OF GAUSSIAN AND LAPLACIAN OF GAUSSIAN
The Gaussian operator is a smoothing filter, which reduce the sensitivity to noise. The function of Gaussian is shown as below.
G(x, y, σ ) = 1 The function of DoG [27] is used to fit the discrete pixels in scale-space. The DoG operator is shown as below.
For a two-dimensional image I(x, y), the response value of Laplacian of Gaussian (LoG) reaches the maximum when the scale σ = √ r. If the circular spot in the grayscale image, black and white are reversed, the response value of LoG reaches a minimum at σ = √ r. The Laplacian operator is shown as below.
The function of LoG [28] is shown as below.
First order partial derivative of the Equation (1) with respect to independent variable σ is shown as below.
When σ → 0, the limit of Equation (6) in scale-space is shown as below.
Usually, DoG is considered as an approximate extreme value of LoG in order to conveniently compute. We also adopt the DoG operator to detect the local feature of image, which effectively accelerates computing.

B. HARRIS CORNER ALGORITHM
Harris Corner algorithm [29] is an extremely simple detecting algorithm. The algorithm adopts the moving window to calculate the gray level change and identify corner, edge, smooth region by the gray difference [30] of adjacent pixels. As the window moves [u,v], the gray change of pixels in the window is described as follows.
The Equation (9) can be expand and approximately computed by Taylor formula as follows.
Calculating the Harris matrix [31] of each pixel in the image, then getting the eigenvalues of characteristic matrix. The two eigenvalues of uniform plane are small or approximate. The eigenvalues of edge are different, one of which is large while the other is small. The eigenvalues of corner both are large or approximately equal. VOLUME 9, 2021

V. THE OVERALL WORKFLOW OF THE PROPOSED ALGORITHM A. SPECIFIC STEPS OF THE PROPOSED ALGORITHM
The proposed algorithm includes seven steps, which are shown as follows.
1) The captured image is read into the program and processed into the gray-scale image. Then adopting normalization operation to make features of image invariant in scale, rotation and affine. 2) Constructing Gaussian scale-space and computing the image by convolution operation, then generating DoG by equation (2). The aim is to reduce noise effect and make features of image more significant by different scales of Gaussian. 3) Judging all pixels' gray values of the image, then using binarization operation by the interval of [minThreshold, maxThreshold]. It is important to filter out the feature of non-defective image for the next processing. 4) According to all pixels' gray values of the image, connecting feature pixels to make large connective domains. Based on binary image, using the findContours() function to extract the connective domain and calculating the center of each connective domain. The function of findContours is designed as below: findContours(image, contours, hierarchy, mode, method, offset=Point()) image-Source. The image is the input data source, which is treated as binary.
contours-Detected contours. Each contour is stored as a vector of points.
hierarchy-Optional output vector, containing information about the image topology.
mode-Contour retrieval mode, which can be set to CV_RETR_EXTERNAL or CV_RETR_LIST or CV_RETR_CCOMP or CV_RETR_TREE. method -Contour approximation method, which can be set to CV_CHAIN_APPROX_NONE or CV_CHAIN_APPROX_SIMPLE or CV_CHAIN _APPROX_TC89_L1 or CV_CHAIN_APPROX_TC 89_KCOS. offset-Optional offset by which every contour point is shifted. 5) Building block area: According to the minDist distance between the close points on the edge of two connective domains, judging whether the close points are classified into one group or not. 6) Calculating the size of each block. If the value of block is greater than the threshold (100), it would be considered as a flaw. 7) Outputting the value of flaws' area and the coordinate of center points in the image.

B. DESCRIPTION OF THE PROPOSED ALGORITHM
According to our proposed algorithm, calculating time complexity of the algorithm step by step as below: If I 2 ( l 1, l 2 ) >minDist #Judge the distance of Connectivity I 2 (l)=I 2 ( l 1, l 2 ) #Coupling 5: For k in I 2 : Grey-scale Map: m * n Normalization: m * n Scales of Gaussian: m * n * kszie Binarization: m * n Constructing Connectivity: m * n Building block area: m * n * d Judging the area by threshold: L In the expression above, m * n indicates the size of image, kszie indicates the size of the filter of Gaussian, d indicates the longest diameter of the block area.
The total time complexity of the algorithm is O (m * n)+O (m * n * kszie) + O (m * n * d).
Because executing the algorithm only need one dimensional array and two dimensional arrays, the total space complexity of our algorithm is O(m * n) + O(n).
Sample 1 with 665 × 1037 PPI, 24-bit RGB and 92.9K was captured by a HD camera. After processing the image of product surface via the proposed algorithm, the effect of flaws on product surface is shown in Figure 6. The red circle in the Figure 6 are flaws on product surface. In order to make the flaws obviously distinguish, we annotate it by drawing straight lines.
Sample 2 with 4032 × 3024 PPI, 24-bit RGB and 1.16M was also captured by a HD camera. After processing the image of product surface via the proposed algorithm, the effect of flaws on product surface, as shown Figure 7 on the right. The red circle in the Figure 7 are flaws on product surface. In order to make the flaws obviously distinguish, we also annotate it by drawing straight lines.   Sample 3 with 662 × 996 PPI, 24-bit RGB and 179K was captured by a HD camera. the effect of flaws on product surface is shown in Figure 8. There are seven flaws on the product surface. The experiment result indicates the flaws are easily distinguished by the proposed algorithm.
In the experiment, we have compared our proposed algorithm with detecting local features algorithms, identifying edge algorithms and extracting texture algorithms. In the aspect of detecting features, the local feature algorithms [3]- [8] extract too many feature points to identify flaws on the product surface of auto parts, as shown in Table 1. Some points of non-defective surface are also extracted as key features, as shown in Sect. 3 Figure 4. In addition, the texture of  flaws on the product surface is very similar to the background texture, so it is very difficult to extract feature of flaws by the extracting texture algorithm [17], [18]. The defective product could not be identified, as shown in Sect. 3 Figure 3. We also experiment to identify flaws on product surface via the identifying edge algorithm [23], [24]. It is invalid to extract the feature of surface image of product, as shown in Sect. 3 Figure 5. So we propose a method for identifying flaws on product surface based on spatial connectivity domain. The method is used to accurately identify a number of flaws on the product surface, as shown in this Sect. Figure 6,7. Moreover, the speed of the proposed algorithm is also suitable to detect product of auto parts in factory assembly line. The lightness or angle of view could affect the detection robustness. It is important to evaluate and calibrate the algorithm under different light conditions. In order to verify VOLUME 9, 2021 the detection robustness of the proposed algorithm, we have changed the lightness and angle of samples to experiment in different lightness or each angle condition, as shown Table 2. We found that when lightness is increased by over 33%, the proposed algorithm is unavailable. But the sample with the increment of lightness ranges from 0 to 33%, our algorithm is available, as shown Figure9. Furthermore, the proposed algorithm is available when the sample was rotated to adjust the arbitrary angle, as shown Figure10.

VII. CONCLUSION
During the production process of inspecting product, the manual processing costs much time and labor force. It is an encountered problem to identify the features of flaw on the product surface. Because the texture of image is very similar, and the intensity of the feature region is consistent with the surrounding area. So it is much more challenging than the traditional algorithms, such as extracting texture algorithms, detecting local feature algorithms and identifying edge algorithms. We have compared Tamura texture analysis method, Gabor wavelet texture extraction method, LBP texture statistical method, AKAZE algorithm, KAZE algorithm, BRISK algorithm, SIFT algorithms, Sobel algorithm, Scharr algorithm, Roberts algorithm. These algorithms are invalid to identify flaws of the product by extracting the image feature of the product surface. So we exploit a method for identifying flaws on product surface based on spatial connectivity domain. The proposed method can accurately identify a number of flaws on the product surface. According to experiments of samples, the effect of identifying flaws on product surface is good. Moreover, the speed of the proposed algorithm is also suitable to inspect the product of auto parts in factory assembly line.