Printed Circuit Boards Defect Detection Method Based on Improved Fully Convolutional Networks

Since printed circuit board (PCB) is the key to ensure the reliability of electronic equipment. Therefore, defect detection for PCB is a basic and necessary work. This paper proposes a PCB defect detection method based on an improved fully convolutional neural networks to detect four types of defects: spurs, mouse bites, short circuits and open circuits. The improved neural networks increase the receptive field and solve the problems of ordinary atrous convolution information loss of continuity by introducing the continuous atrous convolution module. At the same time, we introduce the improved skip layer after the upsampling layer to get the characteristics of multi-scale fusion, improve the resolution of the image. Due to the fewer levels of the whole networks, MobileNet-V2 networks model is adopted in the improved fully convolutional neural networks, which is compared with ResNet-50 and Vgg-16 networks model. Comparison the accuracy of four kinds of PCB classification defects, experimental results show that the improved fully convolutional neural networks based on MobileNet-V2 have an average recognition accuracy of 92.86% for four types of PCB defects, which is better than the ResNet-50 and Vgg-16 networks models, verifying the effectiveness of the PCB defect recognition classification. Moreover, experiments are carried out in a real environment to verify the feasibility of the proposed method.


I. INTRODUCTION
With the rapid development of PCB manufacturing industry, the requirements for PCB process design and quality are getting higher and higher. There are more and more typed of defects on its surface, such as spurs, mouse bites, short circuits and open circuits. The requirements for detection accuracy are also getting higher and higher. Now the common detection methods can't guarantee the quality requirements of PCB and accurately detect the defects on PCB surface in the mass production process. Machine vision is a comprehensive technology used to simulate the partial functions of the human eyes and brain to realize the recognition and detection of objects including mechanical engineering, optical imaging systems, control engineering, image processing, The associate editor coordinating the review of this manuscript and approving it for publication was Wenbing Zhao . lighting systems, sensors, computer technology and so on [1]. The Automated Optical Inspection (AOI) system is based on machine vision, with complex algorithms and comprehensive functions. After years of research and development, it has gradually reached a bottleneck. Due to the difficulty of optimize further the existing cutting-edge AOI system, fewer improvements to the system and the high price, the related research is gradually decreasing. In recent years, the more popular deep learning algorithms have made breakthroughs in the image field, and their performance has been greatly improved, which provides a feasible idea: the deep learning algorithm model can be used to detect PCB bare board defects.
In recent years, researchers have proposed many methods for PCB defect detection. Li et al. [2] added two residual units to the second residual module to improve the network's ability to extract shallow features. Meanwhile, SE Block module was added to the network to improve YOLO v3 network. Zhang et al. [3] combined the existing PCB defect data set with some artificial defect data and affine transformation, and used pre-training deep convolutional neural networks to learn high-level defect identification features. Zhu et al. [4] improved the lighting method by using color light source array lighting and multi-TDICCD splicing camera imaging, and using color space features of color images to determine the defect location of PCB board. Mukesh et al. [5] detected defects of PCB bare boards by image enhancement and standard template generation using standard database, while Particle analysis is used for standard template generation. Gaidhane et al. [6] proposed an effective similarity measurement method for PCB surface defect detection. In the proposed approach, a symmetric matrix is calculated using the companion matrices of two compared images. Chen et al. [7] used VGG-16 and Mobilenet-v2 models for comparative analysis to study a fast and accurate method for wafer structure defect identification and classification.
Although different methods have achieved some research results in specific application scenarios, there are still many problems to be improved. For example, the accuracy of PCB defect classification accuracy and its detection rate. To solve this problem, this paper proposes a PCB defect detection method based on improved fully convolutional neural networks. Continuous atrous convolution and improved skip structure are introduced into the traditional fully convolutional neural network to avoid gridding effect, and the image resolution is improved by feature fusion through upsampling with three different magnifications. We build a PCB defect recognition and classification model based on MobileNet-V2 [8] networks model. Combined with the experimental results, the validity of the proposed networks model for PCB defect identification and classification is verified.

II. ALGORITHM FLOW
The process of PCB defect detection method proposed in this paper is shown in Fig. 1. Charge Coupled Device (CCD) camera is used to collect RGB three-channel PCB images, including standard PCB images without defects and PCB images to be tested with four defect types. In consideration of various non-anti-interference effects, the collected PCB images need to be pre-treatment. After image registration, imabsdiff function differential operated to locate defects [9]. Finally feature extraction and defect classification based on MobileNet-V2 improved fully convolutional neural networks model, so as to realize the whole PCB bare board defect identification [10], [11], [12], [13], [14], [15], [16] algorithm flow.
In the image pre-treatment, the gray transform adopts the weighted average method. The component values of the three channels of RGB color image are taken as a weight respectively for weighted average. Image filtering uses two-dimensional Gaussian distribution as image smoothing filter, and enhances image contrast through logarithm transformation. The Otsu is used for threshold segmentation [17], and the optimal threshold is determined by judging the maximum variance of two parts of pixel values. The normalized cross-correlation template matching method is used for image registration [18], [19] to find the region matching the template image in the searched image. Finally, by using the imabsdiff function differential-operated to locate defects, the defect image is input into the improved fully convolutional neural networks to classify and identify defects.

III. IMPROVED FULLY CONVOLUTIONAL NEURAL NETWORKS MODEL DESIGN
The difficulty of PCB defect detection lies in that the area of the small defect, and the defect characteristics are not obvious without enlarging the picture, that is, it is difficult to be extracted by the classifier, and serious missed detection exists, which cannot meet the practical requirements of the production inspection link. In order to solve the problem of small targets and few features, this paper proposes two improved schemes based on MobileNet-V2 improved fully convolutional neural networks to improve the feature acquisition ability of the trunk networks for small targets. The structure of the whole networks is shown in Fig. 2.
A three-channel PCB image with an image size of 640 × 640 was input into the network model. The model adopted three convolution layers and three pooling layers in feature extraction. Since only focusing and non-focused important information can be extracted during multi-focus VOLUME 10, 2022 image during fusion, three convolution layers and three pooling layers were sufficient to obtain the required feature information. After the compression of the max-pooling, the size of the image becomes 56 × 56 × 256 and enters the full convolutional layer. After continuous atrous convolution expands the receptive field, the size becomes 1 × 1 × 2048, and the size of output and input images is 448×448×4 through upsampling and improved skip image fusion. The probability distribution of focused and unfocused information is obtained by ordinary convolutional neural network in image recognition. The results of fully convolutional neural networks recognition are focused images and unfocused images, and the transmission from image to image can reduce the operation time of the whole network during training.

A. STRUCTURE OF MobileNet-V2 NETWORKS MODEL
Compared to the MobileNet-V2. The convolution kernels of the Vgg-16 and ResNet-50 are all replaced with 3 × 3. In the network testing phase, the three fully connected in the training phase are replaced by three convolutions, so that the fully convolutional network obtained by testing can receive input images of any width or height because there is no restriction of full connections. And ResNet-50 adds residual mapping, however, the number of parameters is as much as 140M, which requires more storage space.
The convolutional part of this paper adopts the inverted residuals in MobileNet-V2 [20], [21], [22]. That is, we firstly use the 1 × 1 convolution to dimensional-raising, then use the 3 × 3 Depthwise(DW) convolution to extract features, and finally use the 1 × 1 convolution to achieve dimensionreduction. Meanwhile, the incentive function adopts ReLU6 function, as shown in Fig. 3 for the convolutional module of MobileNet-V2. Assuming that the size of the input image is x × y × z, the convolution kernel of 1 × 1 is first applied to the image for dimensional-raising processing. The output size of the image is x × y × (tz), where t is the expansion factor, and the image depth becomes tz after dimensional-raising processing. Then through DW convolution of step is s, the image output size becomes x s × y s × (tz). Finally, the convolution kernel 1 × 1 is used to dimension-reduction of the image. The output size of the image is x × y × z , which is the structural mode of the entire inverted residuals.
ReLU activation function is used in the ordinary residuals, while ReLU6 activation function is used in the inverted residuals, because ReLU activation function will cause a lot of losses to low-dimensional characteristic information. The expression of ReLU6 activation function is shown in Equation (1): The image of ReLU6 activation function is shown in Fig. 4:

B. IMPROVED NETWORKS STRUCTURE 1) CONTINUOUS ATROUS CONVOLUTION MODULE
In this paper, continuous atrous convolution module is introduced after the fully convolution layer, as shown in Equation (2). Ordinary atrous convolution is defined as: where f [i] is the input signal, and g[i] is the output signal, h[l] representing the filter of length is L, r corresponding to the dilation rate that we use to sample f [i]. In the standard convolution, r = 1. The so-called atrous convolution is to insert zeros into the standard convolution kernel to expand the receptive field. Compared with traditional convolution, atrous convolution can retain the internal structure of data and compensate for the information loss caused by pooling, but it will lead to the loss of spatial hierarchy and information continuity, thus causing the gridding. For the convolution kernel whose convolution kernel size is k ×k, the obtained expansion filter size is k s ×k s , where Atrous convolution is used to maintain the high resolution of the feature graph in Fully Convolutional Networks (FCN) by replacing the max-pooling or strided convolution layer [23], [24]. Since atrous convolution introduces 0 pixels into the convolution kernel, the actual pixels k s × k s involved, and the calculation of the region is only k*k, the gap between non-0 pixels is r − 1. If k = 3, r = 3, only 9 of the 49 pixels in this area are used for calculation, as shown in Fig. 5. Therefore, when the dilation rate increases with the under-sampling, the local information of the image may be completely lost, which is not conducive to the extraction of image features.
The continuous atrous convolution module proposed in this paper solves the problem of loss of information continuity in ordinary atrous convolution. Taking the three dilation rates [1,2,4] as a group, the features of different scales can be fully extracted by combining atrous convolution with different dilation rates. Suppose there is N convolutional layer with a convolution kernel of size S × S, and its dilation rate is [r i . . . r n ]. The maximum distance between two non-zero pixel values for convolution of consecutive atrous is defined as: (4) where M n = r n , the purpose of definition is to select a reasonable dilation rate to avoid gridding, when M 2 ≤ S, it indicates that the dilation rate is satisfied. For example, when the convolution kernel size is S = 3 and the dilation rate is r = [1, 2, 4], M 2 = 2 satisfies M 2 ≤ S, so the dilation rate selection is effective. When the size of the convolution kernel is S = 3 and the dilation rate is r = [1,2,8], if M 2 = 4 does not satisfy M 2 ≤ S, the selection of the dilation rate will have a gridding. Fig. 6 is a preview of the effect of the continuous atrous convolution module in this paper, where (a), (b) and (c) are a group of continuous atrous convolution with expansion rates r = [1, 2, 4] respectively. The convolution operation completely covers the square area of the receptive field.
The continuous atrous convolution module can use an arbitrary dilation rate in the whole process, thereby naturally expanding the receptive field of the networks without adding additional modules, which increases the receptive field and solves the loss of information continuity for ordinary atrous convolution question. However, one thing to note is that the dilation rate within the group should not have a common factor relationship (such as 1, 3, 5, etc.), otherwise the problem of gridding will still exist.

2) IMPROVED SKIP LAYER
The traditional networks model has 5 convolutional layers and max-pooling layers, so the ordinary skip layer output performs 32 times, 16 times and 8 times upsampling operations respectively, and its upsampling rate is too large, so the effect of the image itself is not ideal. Since the improved fully convolutional neural networks proposed in this paper VOLUME 10, 2022  has three convolutional layers and max-pooling layers, the upsampling operation is reduced to 8 times, 4 times and 2 times. As the magnification of the image itself is reduced, the image definition is increased and the accuracy of subsequent image training is improved. Fig. 7 shows the schematic diagram of the improved skip layer in this paper.
After the upsampling layer, an improved skip layer is introduced to obtain the fea-tures of multi-scale fusion to improve the image resolution. The convolution kernel of 1 × 1 is carried out for the image effect after three-layer max-pooling output, and the number of channels is the same as the result after upsampling. The feature fusion is carried out for the result after three-layer max-pooling, and the size is expanded to the size of the input image through upsampling. Fig. 8 shows the structure diagram of the improved skip layer fusion process.
Assuming that the size of the image is h × w, the size of the image is under-sampling by 8 times through the classification networks MobileNet-V2, and the output image size is h 8 × w 8 × 256. After the output of the two convolutional layers with 2048 convolution kernels, FC1 and FC2, the image size is h 8 × w 8 × 2048, and then it is output through the convolution layer with the number of convolution kernels num_cls, where num_cls is the number of networks classifications. The output image size is h 8 × w 8 × num_cls, and the size of the image is upsampling by 2 times through the transposed convolution, and the output image size is h 4 × w 4 × num_cls, and the 8 times upsampling operation in the improved skip layer is completed. The size of the image obtained by under-sampling 4 times by MaxPooling2 is h 4 × w 4 × 256, and the size of the output image is h 4 × w 4 × num_cls after the convolution layer with the number of convolution kernels num_cls. The 8 times and 4 times upsampling are used for feature fusion and the image size is up-sampling by 2 times, the output size is h 2 × w 2 × num_cls, and the 2 times transposed convolution operation is performed after obtaining 8 times and 4 times the transposed convolution fusion image size. The size of the image obtained by under-sampling 2 times by MaxPooling1 is h 2 × w 2 × 128, and the size of the output image is h 2 × w 2 × num_cls after the convolution layer with the number of convolutional kernels is num_cls. At this time, the size of the output image is fused with 8 times and 4 times transposed convolution is consistent, then the feature fusion of the 2 times transposed convolution image and the 8 times and 4 times transposed convolution images is performed and the image size is upsampling by 2 times, the output image size is h × w × num_cls, and the output image size is the same as the original image size. The whole workflow is the improved skip layer described above.
The improved skip layer reduces the initial transposed convolution operation from 32 times, 16 times and 8 times to 8 times, 4 times and 2 times, because the magnification of the image itself is reduced, and the feature fusion is carried out through transposed convolution of three different magnification rates, which increases the image resolution and improves the accuracy of subsequent image training.

A. PCB DATA SET
In this research, this paper collected nearly 1000 images for the experiment, the original PCB defect dataset http://robotics.pkusz.edu.cn/resources/dataset/ contains about 700 PCB defect images, each PCB image pixel is about 2300 × 2300, therefore, we divided the PCB into nearly 300 parts, each part has 640 × 640 pixels, which is 300 images cover almost every part and area of PCB defects. And selected 200 images as the test set to verify its processing effect in practice.

B. EXPERIMENTAL PLATFORM AND TRAINING PARAMETERS
In this paper, the MobileNet-V2 networks model is used in the improved fully con-volutional neural networks. In the experiment process, this paper adopts the deep learn-ing framework of Tensorflow, and the backbone networks of feature extraction and under-sampling adopts the MobileNet-V2 networks model, the operating system is Windows10, the programming language is Python, and the optimal training parameters are shown in Table 1: Based on the Python software platform, this paper adopts three pre-trained fully convolutional neural networks of MobileNet-V2, ResNet-50 and Vgg-16 respectively on the improved fully convolutional neural networks model to detect and identify the defects of the PCB bare board. Four defect types of spurs, mouse bites, short circuits and open circuits were identified. Fig. 9(a) shows the variation of the training loss value and the validation loss value of the MobileNet-V2 networks model with the number of training iterations. As can be seen from the figure, as the number of iterations increases, the training loss value and the verification loss value gradually decrease, and the final training loss value tends to 0.0241, indicating that the networks is close to convergence at this time. The training accuracy and validation accuracy curve of the MobileNet-V2 networks model are shown in Fig. 9(b). It can be seen from the figure that the training accuracy and verification accuracy generally increase with the increase of the number of iterations, and the final verification accuracy is stable at 0.9286. In order to further verify the detection performance of the improved network model in this paper, precision, recall and F1-Score in the evaluation indicators of the target detection algorithm are selected, and the detection results for the four types of defects are shown in Fig. 10. It can be seen from the figure that with the increase of the confidence interval, precision gradually increases and tends to 1, recall gradually decreases and tends to 0.
In machine learning and statistical classification problems, confusion matrices are visualization tools, especially for supervised learning. Confusion matrix of the four types of PCB defect detection and classification results in this paper is shown in Fig. 11. It can be seen from the figure that the algorithm in this paper can accurately identify four types of PCB defects. The abscissa Predicted is the instance prediction of the PCB defect category, and the ordinate True is Example of actual PCB defect categories.
The experimental results are shown in Fig. 12. The improved fully convolutional neural networks based on MobileNet-V2 can accurately identify defects types.  This paper verifies the ResNet-50 and Vgg-16 [25] pretrained fully convolutional neural networks, compared the accuracy, loss and time-consuming of the improved fully  convolutional neural networks based on MobileNet-V2 proposed in this paper. The comparison data is shown in Table 3. It can be seen that the average accuracy of the algorithm proposed in this paper reaches 92.86%, which is 4% and 6% higher than ResNet-50 and Vgg-16 pre-trained fully convolutional neural networks, the average loss value reaches 0.2188, which is 0.09 and 0.1 lower than ResNet-50 and Vgg-16 pre-trained fully convolutional neural networks, which further verifies the effectiveness of the improved fully convolutional neural networks model based on MobileNet-V2 in PCB defect image classification.
Mobilenet-v2 based convolutional neural network, fully convolutional neural network and the improved convolutional neural network in this paper are compared in the performance of accuracy, loss and time consumption. The comparison data is shown in Table 4. The average accuracy of the proposed algorithm is increased by 6% and 2% compared with Convolutional Neural Network (CNN) and FCN, and the average loss value is decreased by 0.2 and 0.12 compared with CNN and FCN, which verifies the effectiveness of the improved fully convolutional neural networks model in PCB defect images classification.
The above is the comparison between the algorithm in this paper and the standard network. In the literature [2], VOLUME 10, 2022 Li et al. proposed an improved YOLO v3 network PCB defect detection algorithm. In order to more comprehensively evaluate the detection performance of the improved fully convolutional neural networks based on MobileNet-V2 proposed in this paper, the algorithm is compared with the improved YOLO v3 algorithm. Precision, mAP@0.5, Recall and F1-Score in evaluation indicators of the target detection algorithm are selected for comparative experiments. The experimental results are shown in Table 5.
As can be seen from Table 5, compared with the improved YOLO v3 network, the improved fully convolutional neural networks based on MobileNet-V2 proposed in this paper has significantly improved Precision, mAP@0.5, Recall and F1-Score on the PCB defect dataset. Indicating the effectiveness of the improved algorithm in this paper.

C. REAL EXPERIMENT
In this paper, the following experimental platform is built in the experimental area of 3m×2m. The experimental platform is shown in Fig. 13. In order to further verify the feasibility of the algorithm in this paper, practical experiments are carried out. Put the PCB to be tested into the conveyor belt, use the XBOX 360 camera to take pictures of the PCB in transit in turn, and finally import the acquired PCB image to be  tested into the PC computer, and use the improved fully convolutional neural networks to identify defects and select the light source in the experiment. Two 50W LED floodlights are used for lighting.
The actual recognition effect of the PCB is shown in Figure 14. As shown in the figure, regardless of whether the surrounding environment of the PCB is complicated, as long as the defect area is not blocked. The improved fully convolutional neural network proposed in this paper can accurately capture the defect area on the PCB board.
The experimental verification shows that the improved fully convolutional neural network proposed in this paper can accurately identify PCB defects. The experimental process is less time-consuming and less expensive, improves the detection speed and the generalization ability of the system, and can meet the requirements of online inspection in the factory real-time requirements.

V. CONCLUSION
Aiming at the four common types of defects in PCB bare board: spurs, mouse bites, short circuits and open circuits. This paper proposes a PCB defect detection and classification method based on an improved fully convolutional neural networks, combined with image pre-treatment algorithm to enhance image features, and then after image registration and difference operation to extract defect area, collect training data set into the optimized networks. Finally a high classification accuracy of 92.86% is obtained on the test set. Compared with the traditional ResNet-50 and Vgg-16 pre-trained fully convolutional neural networks, CNN and FCN, the improved fully convolutional neural networks based on MobileNet-V2 proposed in this paper, which adds a continuous atrous convolution module and an improved skip layer in structure. The effectiveness of the network model proposed in this paper is verified. By comparing with the improved YOLO v3 network, the effectiveness of the network model proposed in this paper for PCB defect identification and classification is further verified, which has higher stability and accuracy of defect detection. Furthermore, The feasibility of the proposed algorithm is verified on the experimental platform.