Printed Circuit Board Defect Detection Methods Based on Image Processing, Machine Learning and Deep Learning: A Survey

Printed circuit boards (PCBs) are a nearly ubiquitous component of every kind of electronic device. With the rapid development of integrated circuit and semiconductor technology, the size of a PCB can shrink down to a very tiny dimension. Therefore, high-precision and rapid defect detection in PCBs needs to be achieved. This paper reviews various defect detection methods in PCBs by analysing more than 100 related articles from 1990 to 2022. The methodology of how to prepare this overview of the PCB defect detection methods is firstly introduced. Secondly, manual defect detection methods are reviewed briefly. Then, traditional image processing-based, machine learning-based and deep learning-based defect detection methods are discussed in detail. Their algorithms, procedures, performances, advantages and limitations are explained and compared. The additional reviews of this paper are believed to provide more insightful viewpoints, which would help researchers understand current research trends and perform future work related to defect detection.


I. INTRODUCTION
The printed circuit board (PCB) is one of the most vital units in the electronic industry [1]. It plays a key role in electronic devices, mechanically holding up and electrically connecting various electronic parts together. PCBs are used in almost every kind of electronic equipment, from electronic watches, smart phones to computers, communication electronic devices and military weapon systems, as long as integrated circuits and other electronic components are present. Benefitting from the development of integrated circuit and semiconductor technology, the size of electronic device components has shrunk down to a tiny scale [2]. PCBs supporting these components are becoming increasingly complicated, diminutive and delicate. Thus, they must be manufactured at a high quality to meet customer demands.
The associate editor coordinating the review of this manuscript and approving it for publication was Mouloud Denai .
During the Fourth Industrial Revolution (Industry 4.0) [3], PCB manufacturing is facing new challenges and opportunities. The core premise of Industry 4.0 is automated industrial processing with high quality, precision and reliability [4]. Therefore, the producing process of tiny complex PCB boards is required to be more stable and reliable with higher speed, meaning that PCB development must be expeditious [5]. Thus, defect detection in PCBs is crucial. If repeated PCB defects cannot be detected fast and precisely, then many produced boards are likely to be scrapped eventually, which is both wasteful and costly. Currently, achieving real-time, high-precision defect inspection and quality control in PCBs to improve yield and profit is vital for manufacturers [6].
Quality control in the manufacturing process of PCBs is usually challenging because a variety of defects occur inevitably due to mishandling or technical faults. Fig. 1 shows common defects in bare PCBs, such as breakout, open circuit, under-etching, mouse bite, spur, short, spurious copper, overetching and broken hole. For printed circuit board assembly VOLUME 11, 2023 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ All these defects could cause the instability of the board or even damage the entire board. Therefore, an efficient, highly accurate automatic detection module needs to be implemented to inspect diverse defects during the PCB manufacturing process. Previously, basic electric tests, including defect detection, were commonly achieved by manual vision, which requires a few experienced labourers, thus requiring factories to invest heavily in recruiting or training them. However, even the most experienced worker might make mistakes during inspection. Conventional manual inspection was eliminated because components and solder joints are becoming smaller, and possible defects are increasing.
Recently, an automated optical inspection (AOI) technique has been widely used to achieve defect detection on the PCB production line [7]. AOI has numerous advantages over conventional manual inspection. For instance, it is able to scan and detect the board quickly while maintaining high accuracy. During the past decade, although many researchers proposed different image processing methods embedded in the AOI system to make it more efficient and intelligent, the methods could still be classified into three categories: reference comparison, non-reference inspection and hybrid inspection methods [8]. Reference template image-based methods are most widely applied in the manufacturing industry. In this method, the correlation or difference between the template images and the inspected images is acquired to help people estimate defects in PCBs. Furthermore, considering the nonreference approach, a pre-defined mathematical algorithm is proposed to detect PCB defects. The hybrid approach combines reference comparison and non-reference methods.
In addition to traditional image processing methods, machine learning algorithms, such as support vector machines (SVMs) [9], neural networks (NNs) [10], genetic algorithm (GA) [11] and decision tree (DT) [12] have been integrated into the AOI system to improve the detection accuracy and speed. However, image processing operation still needs to be combined with the machine learning algorithm, because the processing of input images is the foundation of subsequent feature extraction and defect classification. Reference images are also required in these proposed detection or classification methods, most of which are used to perform the template matching [13] operation to extract features or locate regions of interest.
Although the AOI system can help factories release a certain number of labourers, PCB industries still need to devote adequate manpower for quality inspection in cooperation with the AOI system. Thus, to increase the detection accuracy and speed, which helps decrease labour costs, many researchers focus on building advanced traditional rule-based image processing or machine vision-based detection algorithms. In recent years, especially since 2012 when AlexNet was proposed [14], some researchers have tried to apply convolutional neural networks (CNNs) [15] to extract features for defect detection in PCBs.
CNN-based models have achieved outstanding results in many visual detection tasks such as image classification, object detection and semantic segmentation compared with traditional feature extracting algorithms. It has the ability to capture defect features precisely without requiring additional information even if shadows or reflections are present. With these obvious strengths, the CNN-based object detection algorithm keeps refreshing historical records in almost all object detection competitions and progressively occupies the mainstream position in the field of object detection [16]. Hence, this review will discuss and compare traditional image processing-based, machine learning-based and CNN-based algorithms in PCB and PCBA defect detection.
In this paper, we present a detailed review based on the state-of-art PCB defect detection works, filling the gap that few related reviews exist. This review paper is designed to cover broader scopes of PCB defect detection methods to provide good suggestion to researchers or any other intended readers with different competency levels.
Initially, the research methodology, including adoption of database, choice of keywords, searching and classification of research articles, analyses of selected articles is presented for readers to have a clear cognition of the source of selected articles. In addition to a brief introduction of outdated manual defect detection methods, the recent advances of traditional image processing-based methods, machine learning-based methods and deep learning-based methods for PCB defect detection are also covered up to 2022, where the advantages and limitations of each method are critically discussed. Furthermore, our review paper describes the principles and implementation steps of each method, and analyses the performances by comparing their detection criterions and metrics. The comprehensive description conducted in our review paper is expected to offer more new insights to related 15922 VOLUME 11, 2023 Q. Ling, N. A. M. Isa: PCB Defect Detection Methods Based on Image Processing, Machine Learning and Deep Learning researchers or readers in understanding the latest trends of PCB defect detection research, hence assisting them to outline appropriate strategic planning for their future works.
The contributions of this review paper of PCB defect detection can be summarised as follows: • Detailed descriptions are given of the systematic methodology used for data collection of related articles, including the research definition, choice of search keywords and online databases, classification and verification processes used for selecting relevant papers and research analyses necessary to summarise the discoveries from existing PCB defect detection works.
• A comprehensive review of PCB defect detection methods in past 20 years covering traditional image-based, machine learning-based and deep learning-based is presented. To the best of the authors' knowledge, few reviews for PCB defect detection methods is published except antique ones. Furthermore, recent related reviews only list some PCB defect detection works mainly focusing on traditional image processing and machine learning, without presentation of deep leaning based approaches.
• Detailed analyses including principles and implementation procedures, are conducted in our review paper to investigate the strengths and limitations of previous works, which was not considered in previous finite review papers.
• The performances of previous works in terms of detection speed, detection precision are also discussed and compared with each other. Especially, our review paper focus on the inference time and speed simultaneously of each PCB defect detection method instead of detection precision simply, as in real PCB related industries, detection speed is a significant factor that must be considered for application and deployment. Previous review papers only paid close attention to detection precision without practicability.
• Open research challenges from different perspectives are elaborated by referring to the detailed analyses performed on the state-of-art PCB defect detection works published in recent years. Meanwhile, the corresponding further research directions which are potential to work out each open research challenge are also unveiled in our review paper.
The comparison between our survey and previous published review papers of PCB defect detection is concluded in 1, showing the improvements of our survey.
The remaining sections of this paper are presented as follows: Section II presents the research methodology used to conduct a systematic literature review of PCB defect detection. Section III introduces the manual inspection method. Section IV discusses the image processing-based defect detection methods. Section V discusses machine learning-based defect detection methods. Section VI discusses deep learning-based defect detection methods. Sections VII and VIII present the open research challenges and future research directions, respectively. Lastly, Section IX concludes this paper. The important abbreviations used throughout the paper are listed in Table 2.

II. RESEARCH METHODOLOGY
This section presents the methodology utilised to review PCB defect detection methods. As shown in Fig. 3, a research framework is presented in four main steps: research definition, paper search, paper verification and research analysis [19].
First step: Research definition Step 1.1: Define the research area. The research areas are PCB defect detection by traditional image processing, machine learning and deep learning methods, as shown in Fig. 4.
Step 1.2: Define the research goal. The research goal is to provide a systematic research scheme and propose ideas to improve the existing PCB defect detection methods.
Step 1.3: Define the research scope. The research scope involves a review of papers on PCB defect detection works, describing their principles and implementation procedures, comparing their performances and discussing their strengths and limitations.
Second step: Paper search  Step 2.1: Define the search terms. The main references were published from 2000 to 2022. Six key terms were used to search related articles, as shown in Fig. 4. All research areas are the main focuses of the review process presented in Section III-VI.
Step 2.2: Search in databases. The online journal databases used for searching research papers based on the search terms are listed as follows: -IEEE Xplore -Web of Science -ScienceDirect -Springer -Wiley -Scopus Step 2.3: Select only English language papers. Only related articles written in English were reserved. A total of 124 articles were selected for the survey.
Third step: Paper verification The articles collected in the second step were thoroughly verified by all co-authors. Only the articles related to PCB defect detection were selected.
Step 3.1: Filter and remove irrelevant papers. Articles that generally do not have any significant relation with PCB defect detection were discarded except for several general articles used to introduce basic concepts of different algorithms.
Step 3.2: Double verification process for each paper. The remaining selected articles were determined if they lie in the areas of focus. In this review, our focused parts were architectures, procedures, detection accuracy, inference time and robustness of a detection algorithm, as shown in Fig. 5. Other outstanding articles were used as extra references, thus promoting the value of this review. The articles were mainly obtained from reputable journals, and top-level conference papers in related fields such as IEEE Conference on Computer Vision and Pattern Recognition (CVPR), European Conference on Computer Vision (ECCV) and Conference and Workshop on Neural Information Processing Systems (NIPS) were selected.
Step 3.3: Classify and store the papers. The gathered articles were divided into 5 specific sub-folders according to their different focus topics: manual inspection, traditional image processing-based, machine learning-based, deep learning-based and other defect inspection. This process was necessary as it ensured the information related to our research was well organised.
Final step: Research analysis Step 4.1: Analyse the selected papers. We analysed the selected articles to acquire important information such as the frameworks and procedures proposed in the PCB defect detection methods, the improvements compared with original algorithms, the detection performance metrics, detected defect types, inference times and limitations.
Step 4.2: Summarise the papers and make critiques for the proposed methods. We listed the detection criterion, detection performances, advantages and limitations of the proposed approaches. Discussion was conducted to systematically analyse the common shortcomings of existing manual inspection methods, traditional image processing-based methods, machine learning algorithm-based methods and CNN-based methods.
Step 4.3: Identify future research directions and further modifications. Finally, we discussed the potential future development directions for PCB defect detection based on the current research trends and open research challenges observed from the current review analyses.

III. MANUAL DEFECT DETECTION METHODS
Before the AOI technique was applied to detect PCB defects, all the electric tests including defect detection of PCB boards were accomplished through manual vision inspection in most manufacturers. Operators always use a magnifying glass or calibrated microscope to determine whether the PCB is failed or qualified, and to determine when a correction operation is necessary [17]. The tools they utilise are visual inspection and qualified experiences. In this way, experienced inspectors can successfully identify flaws and the types of defects precisely.
Another benefit is that this method has low upfront costs and does not require test fixtures. However, this method highly depends on the skill of operators, who are expected to be practised. This means that factories must spend a large amount of money for labour recruitment. However, even the most experienced expert can make detection mistakes because of his subjective estimation [20]. The disadvantages of this method include high long-term cost, discontinuous defect detection and difficulty in data collection [21]. At present, as PCB production continues to increase and the size of PCB components continues to decrease, many tiny possible defects are emerging. Thus, this outdated manual inspection method, with its low detection speed and efficiency, has become increasingly impractical and has been eliminated [18].

IV. IMAGE PROCESSING-BASED DEFECT DETECTION
Different from manual inspection, the AOI system can eliminate subjective errors and provide a quick, quantitative estimation. This system can work all the time as it does not get tired and burned out, allowing a factory to save a large amount of labour costs. Recently, to promote detection accuracy and speed, which means better quality and lower costs, many researchers have introduced machine vision technique using traditional rule-based image processing method to detect PCB defects. In this section, image processing-based defect detection methods or algorithms for PCBs or PCBA are described.

A. TECHNIQUES FOCUSING ON SURFACE DEFECTS IN PCB
Pal et al. [22] applied the image subtraction method using machine vision to detect bare PCB defects in 2011. This image subtraction approach [8], [23] which is categorised as a reference comparison method, was performed in several steps. Firstly, a reference image has to be obtained among selected good templates. Then, the reference template must be buffered to make it acceptable for subtraction. Next, the inspected image is loaded. To obtain the PCB defects or errors, pixel-by-pixel XOR logic operation, which is also referred to as image subtraction, was implemented between the reference image and inspected image. The resultant image acquired from this subtraction operation should exhibit several defects if the inspected board was defective.
The authors in [22] achieved typical defect detection such as over-etchings (i.e. opens), under-etchings (i.e. shorts) and holes. By subsequent image processing procedures, the precise positions and sizes of defects also were given. However, this method reached its limitations because the reference image needs to have the same size and orientation as the inspected image in terms of pixels, which is very hard to be implemented in real applications. A tiny shift in pixel can cause many false positive targets because the image subtraction is achieved through logical operation pixel-bypixel, which will give rise to huge financial loss for PCB manufacturers. Moreover, acquiring a real qualified reference image is challenging and time consuming as the illumination environment is always changing all the time. Firms need to prepare different templates corresponding to different PCB boards. Subjected to these prerequisites, the accuracy of this method was not desirable. Missed inspection for bare PCB defects, including open, short, incorrect dimensions and missing conductor occurs frequently by this method. Besides, achieving a high processing speed is impossible for this method due to several additional image processing operations such as image thresholding and image particle analysis.
Ma [24] proposed a similar method to detect defects in bare PCBs using the comparison between the reference image and inspected image. The modification was the standard image was an average of a series of qualified PCBs instead of using only one image. The histogram analysis of the difference  image was used to determine an appropriate threshold to distinguish potential defect regions. Furthermore, a filter based on target region contours was proposed to capture the real defects from these possible regions, which was reported to outperform those of depending on simple image thresholding. A modified region growing method was applied to obtain the complete zones of the real defects. The defect types were determined by the changing times of the peripheral boundary's grey value.
Method [24] could detect 90% of the defects, including spurious copper, spur, short, mouse bite and open circuit in 100 target images. Compared with [22], this method addressed the template offset problem by averaging and updating the standard image dynamically. In addition, more image processing operations including bilateral filtering, histogram analysis, region contours filter and region growing were applied to improve the detection precision. So this method can recognize the actual type of defects instead of over-etchings and under-etchings only. Nevertheless, in the operation of updating standard image, the choice of temporal distance between frames becomes a tricky question, which depends on the size and speed of the moving object. Furthermore, many processing procedures cost huge computation, resulting in a slow speed, especially for high resolution images. Another limitation of this method is that the recognition rate for defect types is not desirable. The information of region contours is not enough to decide the type as some defects have similar outlines and sizes. The colour change of surrounding environment could be considered as an important clue to improve the recognition rate.
Melnyk and Tushnytskyy [25] proposed a clustering-based comparison method to detect defects in bare PCBs. A small shift of routing paths and contacts would be considered defects in the traditional subtraction method. To avoid this problem, they applied clustering to the standard and inspected images before the comparison. When the etalon image and defect images were clustered, Equation (1) was applied to obtain the PCB image with marked defects, where I r (x, y), I e (x, y), I d (x, y) are the pixel intensity of the resulting image, the etalon and the defect PCB image, respectively. Tol is a tolerance value to control a difference between an etalon PCB and the controlled sample.
The authors in [25] confirmed the robustness of their method by setting different tolerances such as 50, 150 and 200. The results proved that a comparison of clustered PCB images can ignore small shifts of routing paths and contacts and simultaneously capture real defects. Apart from common limitations of image comparison method presented previously, the difficulties of this method lie in the chosen tolerance and the cluster parameters which need experienced users and many attempts. Improper tolerance can miss real defects as PCB defects always occupy tiny areas in a high resolution image. Hence, this method was not generalized and not convincing in some cases, and the final decision should be made by an expert, which is time consuming and laboured for real application.
Instead of concentrating on the defects directly, Wenbin and Su [26] focused on the detection and localisation of circular holes on PCBs, which are vital for positioning or defect detection. In this work, a Canny detector was applied to create the corresponding edge maps [27]. Then, the author filtered these short arcs by using a threshold called the step of simpler and faster edge thinning and cleaning. Next, the pre-processed image was transformed by using distance transform to acquire the distance space map. In order to position the centres of these holes correctly, two more steps for searching hole centres-a process called rapid circle centre search-were proposed in the distance space.
The method proposed in [26] was compared with an improved circle Hough Transform (CHT) based method reported by Atherton and Kerbyson [28] to prove its better results. The execution times for Atherton's method and the proposed method were 2.4196 and 1.3420 seconds, respectively, and the F-scores were 0.8083 and 0.9167, respectively. This paper provides a method to position holes on a PCB with a good detection performance than traditional hole detection algorithms. However, this method is still time consuming for real coloured samples, and a balance between speed and accuracy should be considered to choose a suitable threshold to thin the edges. Moreover, the significance of this paper is limited as it only focused on PCB holes, which merely can help detect or position defects related with holes in PCBs. In the real environment, it is impossible for workers to predict the defect types to choose specified detection method. So this method is not able to be deployed commonly in the real industry with high efficiency.
Recently, Jakkrit and Jakkree [29] proposed an algorithm that can detect and classify 14 PCB defect types [8]. In this algorithm, colour images were firstly converted into greyscale images, and median filtering was used to mitigate the noise. Then, the median-filtered images were scaled, rotated and translated by the operation of affine transform. Fuzzy c-means clustering was applied to segment images before image subtraction for defect detection. In the defect detection stage, image subtraction was applied to obtain the difference between the segmented and the template images. The results for the subtraction were negative, positive and zero, which indicated excess copper, missing copper or non-defective PCB, respectively. In addition, arithmetic and logic operations such as CHT, morphological reconstruction (MR), and connected component labelling (CCL) were used in defect categorisation. The CHT algorithm transformed a variety of feature points from the image space into a series of accumulated votes in parameter space to detect and classify circular patterns. The MR algorithm categorised features such as holes, pads and traces of the segmented images successively, and then converted the connected background pixels into foreground pixels until the object boundaries were reached. CCL [30] scanned the segmented image, divided the pixels into components based on their connectivity and assigned labels to each pixel until the boundary is reached.
The algorithm proposed in [29] achieved 100% defect detection and 99.05% defect classification accuracies for all 14 known PCB defect types and determined defect characteristics such as location, area and nature, showing noticeable superiority than previous works. However, the input images used in this paper were PCB circuit layout schemes in black and white instead of real PCB images. Even for these low resolution images, the computation was large (i.e. from 1.31 seconds to 5.56 seconds per image). The reason is that from [29], about 20 steps were still needed after the pre-processing, segmentation, subtraction and operations of CHT, MR, CCL. This condition means that the computer processing unit must be upgraded to achieve an acceptable detection speed for the real PCB images. Besides, the processing speed also depended on the image segmentation algorithms used including global thresholding, adaptive thresholding, k-means clustering and fuzzy c-means clustering. This indicates that this approach is not efficient and generalized for fast PCB defect detection because users have to decide the type of image segmentation algorithm used for different applications. VOLUME 11, 2023

B. TECHNIQUES FOCUSING ON COMPONENT AND SOLDER JOINT DEFECTS
Unlike previous methods on detecting surface defects in PCBs, Sundaraj [23] used the background subtraction method to inspect the missing or misaligned components in PCBA. The author provided a solution that was able to monitor a larger region in a PCB at any moment. This automatic visual inspection system was considered efficient at decreasing the inspection time and increasing the handling capacity. In this work, Mixture-of-Gaussian distribution [31] was chosen for background modelling. Then, three stages-background learning, parameter estimation and pixel classification-were implemented to build a Gaussian density function. During the experiment, the non-defective PCBA was firstly fed to the learning process to acquire the template or background image by capturing about 100 frames of the PCBA board. Subsequently, the image of a defective PCBA with missing or misaligned components was captured. The defective image was subtracted from the background image to obtain a binary mask that presents foreground and background pixels. A large area of foreground pixels indicated some kind of defect on the position.
The method reported in [23] could detect missing or misaligned capacitor and chip with an accuracy of more than 90%. However, this method has several limitations. The process of background learning to build a reference image cost huge time (about 3 to 4 seconds), which makes this technique not suitable for real-time detection. Similar to previous works, image subtraction requires both the reference image and tested image must be placed in the same orientation precisely, which is trivial to be achieved in PCB inspection industry. Besides, the accuracy highly depends on hardware such as a rigid optical table, an advanced camera with a macro zoom lens and a fixed lighting with a suitable illumination angle. Furthermore, this system was captious for inspected images. It failed to obtain good results on the component whose colour was the same as that of the background. Same colour presents same or similar pixel values, which will be considered as background and missed in the subtraction operation.
Conventional normalised cross-correlation (NCC) was used to detect defects by comparing the reference image and defect image in [32], but this method is time consuming as it computes the NCC depending on two-dimensional images. Therefore, Annaby et al. [33] proposed an improved normalised cross-correlation (INCC) method to detect integrated circuit (IC) defects such as misaligned components in PCBs. As to the standard image and defect image, considering two corresponding pixels that they contain, blocks with the same size, in which the top left corners are the two pixels, were created. Then, these two-dimensional block images were converted into one-dimensional feature vectors in the matching process. Next, these vectors were augmented by spatial statistical features following discrete cosine transform. After this process, the feature descriptors were obtained. As in the classical NCC method, the correlation coefficient between the corresponding one-dimensional feature descriptors in the standard and defect images was then calculated. A pixel with a correlation coefficient smaller than a specific threshold is classified as a defect pixel in the testing image. Otherwise, the pixel is a defect-free pixel.
The INCC algorithm proposed in [33] was compared with other popular methods, including conventional NCC, Yoo2010 [34] and Fouda2015 [35] using PCB image datasets. The results proved that INCC performed better than others. INCC have fewer false alarms than other two methods, and the defect regions are much more coincident than the methods whose defect regions are smaller than those of the standard model. INCC detected defect regions at least four times faster than NCC when the image size was bigger than 400 × 500. In addition, this method was robust to the changing illumination environment. However, this method has own limitations. Firstly, it was proved to be useful on images contained only one missing or defective component, which is not suitable for real PCB industry. Secondly, although it cut down the computation cost compared with traditional NCC method, it was still inefficient. The core theory of INCC is NCC calculation represented by the dot product of two vectors transformed from standard and detection image, which must cost huge computation. For a large image, it takes several or even more than 10 seconds during the detection process. Hence, it is not able to meet the real-time demand for industry production testing.
Hassanin et al. [36] developed an automatic defect detection method in PCBs on the basis of speeded-up robust feature extraction (SURF) [37] technology and other image processing operations. The reference image and tested image, which had been transformed into binary images, should be pre-processed by a spatial smoothing filter and then the median filter to remove noise. Then, to eliminate the rotation and translation shifts between them, image alignment was performed by using SURF. The rotation angle and translation distance of misalignment can be calculated by SURF and geometric correction was applied depending on the SURF results. After pre-processing, the images were segmented to obtain the foreground pixels using the Otsu thresholding method. The XOR operation was performed to compare the difference between the segmented template image with the segmented inspection image to obtain the defects. Dilation operation was also implemented to reshape the defect regions. Finally, the features of the subtraction image were extracted by SURF, and the feature datasets of each standard component were combined into a matrix. The Euclidean distance between individual defect features with the known components in the feature dataset was evaluated to identify the defect type.
The work in [36] presented statistical characterisation such as the equal error rate estimated based on correct and incorrect matching results and the area under receiver operating characteristic curve (AROC) to prove the good performance. The introduce of SURF given this method noticeable improvement in detection accuracy. In fact, this method is far from real-time detection. The inference time for one image can reach 6-7 seconds, of which SURF register processing takes more than 2 seconds. Other image processing operations such as dilation, binarization and XOR comparison also consume a lot of time. Furthermore, SURF algorithm deeply depends on the gradient direction from local pixels, which may lead to inaccurate main registration direction. For tiny surface mounted PCBs, large amount of components crowd on the surface to result in a more complicated feature space. Only local pixel information instead of global contexts may not able to correct image shifts and extract enough features, which will give rise to a bad detection performance.
According to previous description, traditional image processing-based PCB defect detection works are tabulated in Table 3. Although most of them achieved PCB defect detection successfully, there are large room for improvements. In traditional methods, feature extraction mainly relies on artificially designed extractors, which requires professional knowledge and complex parameter adjustment process. Meanwhile, each method is targeted at specific applications, with poor generalization ability and robustness. The main limitations can be summarised as follows: • Most traditional machine vision-based methods for defect detection in PCBs used image subtraction or image matching to obtain defect regions. This means the qualified template image is essential. The general problem they must face is that a perfect reference image is always very difficult to acquire. For different PCBs, different reference images have to be prepared, which is time consuming and expensive.
• The detection performance highly depends on the quality of the reference image. Hence, they are not robust to the changing environment such as illumination and noise. Even small shadows can give rise to false alarms. Besides, they are not generalized to different samples. For components whose colour are similar with background or surroundings, the detection accuracy could be terrible.
• Both of them did not reach real-time detection at all, especially for high resolution inspection images from VOLUME 11, 2023 real PCB industry. Many additional image processing procedures except image subtraction are necessary to obtain a high accuracy, which inevitably decrease the inference speed.
• The size and orientation between reference and inspection image must be same in pixels entirely, which commonly needs highly precise calibration equipment and trained manpower. Some works have introduced novel approaches such as averaging inference image, clustering pre-processing and SURF to eliminate the affection caused by image shifts, but the whole detection algorithm get more complex and entailed larger computation costs and more parameter choices than before.
It is difficult for traditional image processing to build a complete rule to achieve automatic detection. Sometimes there is a complex combination of various algorithms with too many parameters and manual adjustment. However, seeking the right and suitable parameters is too tedious. One possible solution may be machine learning that computers learn a set of automatic judgment criteria from enough data by extracting a certain number of deep features. Then, the computer will select a best ''hypothesis'' from the hypothesis set through what it has learned. Hence, the basic image processing operations such as edge detection, morphological analysis, Hough transform, various image thresholding techniques and other feature representation techniques, can be used as a front end for machine learning classifiers (i.e. SVM [9], K-Nearest Neighbor [38], NNs [10]) to construct more in-depth detections. The integration of traditional image processing and machine learning can build a more robust detection algorithm with high inference accuracy and speed than single one of them.

V. MACHINE LEARNING-BASED DEFECT DETECTION
Image subtraction-based traditional image processing methods can achieve high detection accuracy for PCB defects. However, using these methods is always time consuming, and many reference images are essential and can affect the detection results. Many researchers have proposed several machine learning-based algorithms to overcome these difficulties. Many classical machine learning algorithms have been developed and used in PCB defect inspection. In this section, several machine learning algorithm-based methods that mainly focus on solder joints and components are introduced and discussed.
Yun et al. [39] used SVMs and a tiered circular illumination technique to inspect the PCB solder joints. The illumination equipment consists of a Charge-coupled Device (CCD) camera with three circular lamps whose colours were blue, red and green. The solder joint surface has the same nature as the mirror surface. Three lamps illuminate the surface of solder joints at different angles, and then the CCD camera can receive reflected light with different intensity patterns due to the difference in surface slope. The visual information of the solder joints surface can be inferred from the highlighted patterns and images. Then, six characteristic features based on average intensity value and percentage of highlights of images from three colour frames were extracted. Finally, each solder region represented by a six-dimensional feature vector was fed into an SVM layer that consisted of four SVM classifiers to output one of several pre-defined types, including excessive, good, insufficient and no solder.
In [39], 402 solder joints were collected to train and test the model. All the insufficient and no solder defects were detected correctly, and for the excessive class and good class, the detection rates were 96.07% and 98%, respectively. This model was compared with other methods, including K-means classifier [40] and backpropagation (BP) classifier [41], to show its better performance. However, this paper just achieved solder joints classification without localisation as SVM is a classifier only. The solder joint images with a size of 50 × 100 pixels instead of whole PCB images were fed into the model. This means that in real PCB defect applications, solder joints images must be extracted firstly, which is a tedious and inefficient process. Moreover, how to choose the hyperparameter C and kernel in SVM is a tricky problem. Meanwhile, the SVM is difficult to be implemented for large-scale training samples, because it uses quadratic programming to solve the support vector, in which the calculation of M-order matrix will be designed. When the order of matrix is large, it will consume huge machine memory and operation time. In addition, this method required complex three-colour circular illumination system which is not easy to operate.
Ko and Cho [42] combined an Learning Vector Quantization (LVQ) neural network [43] and fuzzy logic [44] scheme to inspect solder joints. The solder images were also obtained by a three-colour circular illumination system. Firstly, the obtained solder images were divided into three sub-images (right, centre and left columns in the longitudinal direction of a solder joint), which were each fed into three LVQ classifiers. The purpose of this operation was to decrease the burden and computation, as increased class number decreases the classification performance markedly [45], [46]. The output values of nodes for each classifier were marked from 1 to 10 for right and left columns, and from 1 to 8 for the centre column based on previous studies [47], [48]. Then, the input images were labelled by an expert into five classes based on their experience. Compared with the linguistic approach, the LVQ classifier has difficulty distinctly categorising complex boundaries among various categories. Thus, a fuzzy rule was proposed based on experts' knowledge to obtain better classification results.
The performance of this method in [42] was tested by using 96 images. The total success rate achieved 95.83% among five solder joints types, which was much higher than that of the original LVQ classifier. Similarly, limited by the computing power, this method also got single solder joint images instead of entire PCB images as inputs, which is definitely not suitable for real PCB industry. A complex three-colour circular illumination system was implemented to obtain color patterns as well. Besides, this method was neither a fully automatic nor efficient detection system, because it highly depended on the expert's experience to make a decision in the process of fuzzy logic scheme. Furthermore, this work only classified the solder joint types without localisation yet because of the original limitation of classifiers.
Belbachir et al. [49] proposed a PCB defect inspection system using wavelet transform (WVT) [50] and multi-layer perceptron (MLP) [51] NNs. Complicated illumination lamps were replaced by a common lighting source. The CCD camera was moved instead of the PCBs so that the system was more flexible and could be placed on an automatic production line. A databank that stored all the components and corresponding defect images to detect was built. A reference board was obtained by assembling images of the components from the database based on the circuit layout. A defective board was obtained by overlapping the defect image with the qualified images of components. For the training path, a series of NNs associated with specific components was trained using a set of images from the databank to recognise the corresponding defects that could appear. For the testing path, template matching was applied to identify the reference point, where the CCD was moved to frame the region of interest (RoI). Then, the WVT was implemented on each captured image to extract features, which were fed into a corresponding trained NN. The outputs of the NN displayed the defect class.
With the system in [49], the authors achieved missing component defect classification and faulty solder joint defect classification. Compared with previous works, this method did not require complex illumination system at all. However, a database storing all the components and corresponding defect images must be constructed firstly, which has to take large amount of time and labour. The training dataset was synthesized by components in the database according to the given circuit scheme. The defective circuits (i.e. missing component) could be obtained by overlapping the component with an area having the same color as the background. All of these complicated pre-processing operations make the detection not easy to be deployed in real PCB defect inspection industry. In addition, template matching was implemented in this method leading to the delay of detection speed. More concrete classification metrics and comparison with other methods were not exhibited in the paper.
On the basis of MLP NN and geometric-wavelet (GW) feature extraction, Acciani et al. [52] presented a novel solder joint defects inspection method. The method consists of four steps: image acquisition, image pre-processing, feature extraction and classification. The image that contained more than one board was acquired by a CCD camera and then processed by three image segmentations. To extract the target central board, template matching with a real reference image was applied using the horizontal energy and the vertical energy vectors of a greyscale image. Then, the central board from the previous procedure was matched with a real IC image to obtain the IC region in the target image. Similarly, the solder joint images were acquired by template matching using the corresponding reference image. In the feature extraction step, geometric features [53] and wavelet features [54], [55] were used together to form the third feature type named GW features. Finally, these features were sent into the MLP NNs to obtain the classification results.
The method in [52] achieved overall 98.8% classification rate among five solder joint defect types, which was bigger than that of the method that used single geometric feature [56] or wavelet feature. The performance of this method was also progressive compared with that of the LVQ NN with GW features. However, this method used reference PCB images to get target solder joints through several image matching steps, which represents a mass of computations, causing a low detection speed. Obviously, it must have the limitations same as previous traditional image processing-based methods that require reference images. Besides, it is very difficult for MLP to select the number of hidden nodes. The authors applied n-fold cross-validation strategy to decide the hidden neurons without regard to the time cost. Furthermore, only defects classification without localisation was executed in this work.
Crispin and Rankov [57] achieved PCB component detection by integrating GA and template matching. A generalised template of the resistor was formed by linearly combining six template images of the same size to average their statistical variation of grey-level intensity values. The common operation of template matching involves sliding the template image on every pixel in the target image to calculate the similarity using normalised NCC, which is computationally time consuming. To reduce the computation, the authors applied Canny [58] edge filtering algorithm to the target image so that the search was constrained to edge positions to decrease the sliding space. Histogram equalisation was applied before edge detection and forming template images. Then, GA was used to search template matches. As GA is skilled at providing a single solution, the author designed a mechanism to allow it to capture more targets. When a region in the target image was considered matched, the position was replaced by an inverse template image so that future correlation values in this position were very poor.
The method proposed in [57] achieved resistors detection successfully, and the running time was about 39.5 s with a population size of 160, recombination percentage of 75% and mutation percentage of 35%. Reference images were required in this method. Template matching or NCC was applied in this method, indication a huge computation cost and low detection efficiency. Moreover, the selection of parameters for genetic algorithm seriously affects the quality of the solution, but at present the choice of these parameters is complicated and mostly depends on experience, resulting in a long processing time. Besides, it is easy for GA to fall into precocity situation and to converge to the local optimal solution, which means the detection ability of the algorithm for new PCB data space is limited.
Cai et al. [59] formulated the solder joint inspection issue as an optimisation problem using robust principle component analysis (RPCA) [60], [61]. IC solder joint images were obtained using a three-colour circular illumination system similar as that in [39] and [42]. On the basis of RPCA theory, the hue channel of a colour solder joint image, which contained major information for colour perception [62], was decomposed into a low-rank component and an error component. An appearance model was built by averaging low-rank components of qualified solder joints images. Then, according to prior human knowledge and appearance, defect score was defined to evaluate the quality of inspected images. A discrimination scheme was applied to make the final decision for a qualified or unqualified image by setting a threshold. The threshold was determined by the maximal value among the defect scores obtained at the iterative stage of building the appearance model.
In the work [59], the best performance of the inspection of unqualified solder joints was achieved when at least 350 qualified solder joint images were used to build the appearance model. In that case, the precision, recall and F-measure were 95.65%, 100% and 97.77%, respectively. However, this method only achieved qualified and unqualified classification without localisation. In addition, the building of an appearance model was essential in this method, resulting in a very slow speed because of the large computation costs of singular value decomposition for the data matrix at each iteration in the RPCA process. It is sensitive to very small rotations, as this method was based on the hypothesis that solder joint images of the same type have the same size without rotation. Furthermore, a complex three-colour illumination system was required, and prior human knowledge was in incorporated into the defect score to achieve better inspection performance. All of these limitations indicate that this method is not generalized and not suitable for real PCB inspection applications.
Hongwei et al. [63] used improved AdaBoost [64] and DT to classify six types of solder joint defects. The whole inspection process consisted of the training stage and the test stage. In the training stage, the solder joint image was firstly divided into several sub-regions, which had different colour distributions for defective solders. 30 geometric features based on three colours were extracted in each subregion. Then, an improved AdaBoost was proposed to seek three necessary optimal features according to their classification performances. Finally, the classifier was built for each sub-region after training. In the detection stage, three steps (sub-region division, critical feature extraction and classification for regions) were executed based on the training results. For defect diagnosis, a new DT that combines classification and regression tree was implemented to obtain the final defect class.
The method proposed in [63] achieved an overall classification rate of 97.3% without localisation. The average time for one chip was 8.6 milliseconds, which was slower than image comparison method. In this method, only the chip images with a small size instead of whole images were fed into the classification algorithm, and one component contained only one solder joint defect. Thus, this method will be still time consuming when detecting an entire PCB image with various types of components. Similar to the limitation of previous works, a three-colour hemispherical circular illumination was required to infer the characteristics of the solder shape. The improved AdaBoost was used in this work to select fewer but optimal features. But two thresholds for each weak classifier were introduced and a threshold of relevance information was also applied to guarantee the selected classifier brought new information. Extra threshold parameters caused more computation cost, resulting in a slow classification speed.
Wu et al. [65] from the same group in [63], achieved solder joint classification using Bayes and SVM. Similarly, a three-colour illumination system was used to obtain solder joint images. Six colour features and one template matching feature were acquired in the component body and solder pad, respectively. Then, feature selection on the basis of information gain was used to identify the optimal features from these 14 features. The selected features were fed into a Bayesian classifier [66] to determine whether the solder was qualified or unqualified. For the unqualified solder joints, several SVM classifiers were applied to obtain the defect type.
In the work [65], the best performance of 100% classification rate was achieved using 7 features for both Bayesian and SVM. However, this method was a two-stage classification algorithm, including Bayesian for binary classification and SVM for multi-classification. The detection focused on only one component image with one defect instead of a PCB board. Besides, the template matching was used by calculating the NCC between template and search images. Due to previous limitations, it must be huge computation, especially for a whole real PCB image. Similarly, this method only accomplished defects classification without positioning yet.
Different from [65], the same authors, Wu et al. tried to combine BP network and GA to inspect solder joints in [67]. The processes of solder joint images acquisition and features extraction were the same as in [65]. Differently, GA was used to select 6 appropriate features, which were sent into the BP [68] network for training by calculating mean square error.
The overall classification rate of 98.46% for 6 types of solder joints was achieved by the algorithm proposed in [67]. This method was compared with DT and SVM to prove its good performance. Same as [63] and [65], a three-colour hemispherical circular illumination was necessary in this method, and the detection focused on only one component image with one defect instead of a PCB board. Moreover, the template matching was also used by calculating the NCC between template and search images. Thus, this method was still distant from real-time detection. In addition, the classification rates for cold solder were not high enough at only about 90%.
Wu and Zhang et al. [69] used colour grads and Boolean rules to inspect solder joint defects. The solder joint images were acquired by the same illumination as [63], [65], and [67]. On the basis of the colour distribution and the solder land size, several regions were defined in the solder joint image. Then, the area features, the barycentre features, the distribution features and the colour grads' features were defined and extracted. Models of 8 different solder joint types, namely, acceptable solder joint, lacked solder, excessive solder, pseudo-solder, component shifted, component lift and tombstone, no solder and missed component, were built according to their appearances and previous feature analysis. The relationship between the solder joint types and regions was derived, and Boolean rules were built to classify solder joints.
The method proposed in [69] achieved an average classification rate of 97.7%, which was higher than that of other approaches in [52], [70], [71], and [72]. However, the prerequisite is that judging criteria must be constructed according to the solder joints' appearances and various feature analysis. This process deeply depends on the expert experience and prior knowledge, which is time consuming inevitably. Thus, this method is not suitable for large-scale applications in real PCB inspection industry. Furthermore, the requirement for illumination system is critical to the construction of rules as different illumination situation will bring large errors into the forming of criteria. The running time reached 11 seconds for one PCB, which was also much faster than other references in the paper, but it was still far from real-time detection.
Li et al. [73] proposed a semantic segmentation method to inspect components in the PCB by using a random forest pixel classifier. In this work, depth images of PCBs were used as inputs due to its high robustness to the influence of the illumination and texture of the target [74]. Depth images were applied to detect human body pose [75], [76], [77] and hand gesture [78], [79], [80], [81] by pixel classification approaches. A three-dimensional model of the PCB was used to synthesise the training set by computer graphic rendering, which contained the depth images and corresponding colourlabelled images. Pixel samples in the depth images were selected randomly, and their depth difference features were extracted. Pixel classification labels were extracted from the colour-labelled images.
Then, the difference features were used to train a random decision forest classifier [82], [83], which built a mapping relation between the difference features and the pixel classification labels. Finally, the semantic segmentation of PCB components was achieved by the trained random forest pixel classifier. Missing or misplaced components were also detected by this method. The prediction of the depth images with missing or misplaced components was compared with the colour-labelled images of qualified PCB. Then, the missing or misplaced component can be detected by calculating and analysing the accuracies of each component.
For the method in [73], the overall component recognition accuracies were 98.96% and 83.64% on the synthetic and real depth images, respectively. This method performed poorly for real PCB depth images, maybe because the real PCB images have more complicated layout and various noises, which can make the random forest pixel classifier overfitting. Additionally, this method could only classify defective or non-defective caused by missing or misplaced without the exact defect types. Another limitation is that several experiments must be performed to choose the optimal training parameters and random seed for the random forest classifier. Hence, this method must be ineffective for real component defects recognition.
Machine learning algorithm-based defect detection works are tabulated in Table 4. Most of them focused on solder joints or components defects detection and achieved good performances successfully. However, large gaps between them with real PCB inspection application in the industry still exist. In machine learning-based methods, features sent into the machine learning algorithms are mainly extracted by traditional image processing operations such as edge detection, morphological processing, various image thresholding techniques. Hence, the detection performance of machine learning-based methods deeply depends on the quality of previous extracted features. Meanwhile, each method was designed for special applications, indicating a poor generalization ability and robustness. The main limitations can be summarised as follows: • All the proposed methods only achieved classification without localisation. Then main reason is that most machine learning algorithms were developed for classification issues. The localisation of defects can be achieved by adding several image post-processing operations.
• Image pre-processing were required before the final classification process, and some methods required a qualified template to locate the reference positions or regions of interest. Similar to traditional image processing-based methods, they also have difficulty acquiring reference images, which is time consuming and complicated. Their performances were deeply affected by the reference image and prior image processing.
• Another common limitation is that some of them used a complex three-colour illumination system to obtain target images with abundant features, which was hard to implement and thus expensive.
• Due to the influence by many image processing steps and the choices of various parameters in machine leaning algorithms, all of these methods cannot perform real-time inspection of a PCB board automatically, although some of them achieved high-speed inspection for one component. Obviously, machine learning-based PCB defects detection methods cannot discard traditional image processing as a VOLUME 11, 2023 front end. We still need to identify and manually code application features based on data types (i.e. pixel value, shape, orientation). Moreover, many machine learning methods cannot handle large-scale data rapidly. Some traditional machine learning algorithms need to solve constrained optimization problems or complex matrix calculations, leading to high time complexity. Some algorithms need to calculate the distance between all sample points, which have high spacetime complexity. More importantly, many traditional learning algorithms are difficult to combine with Graphic Processing Unit (GPU) parallel computation. So they cannot deal with large-scale data efficiently such as real complicated PCB images with high resolution.
To overcome these limitations, deep learning-based algorithms is a good choice. Various CNN models have been used in object detection, image segmentation etc. with desirable detection accuracy and speed simultaneously. Features are not extracted by manual designed extractors, but extracted by CNNs automatically. In addition, they are robust to the environment and noise. Thus, PCB defects inspection also can be considered as an object detection issue in nature, which is possible to be overcome by deep learning methods.

VI. DEEP LEARNING-BASED DEFECT DETECTION
Recently, deep learning-based methods, especially CNNs, have been widely used in image processing [84], object detection and segmentation [85]. Unlike traditional machine vision methods, CNN-based approaches are able to automatically extract image features, simplifying the image pre-processing process so that the detection accuracy and speed can be promoted effectively. In addition, CNN-based methods are robust to the environment and noise. Even though shadows or reflections exist, significant object detection outputs can still be achieved because of these methods' multi-level feature extraction ability. With these advantages, CNN-based object detection algorithms have outperformed competing algorithms on different datasets and have become the primary driving force for the development of object detection. Therefore, deep learning algorithms have attracted the attention of researchers for PCB defect detection and achieved good detection performance.
The most frequently used and modified CNNs to detect PCB defects are Faster Region-based Convolutional Network (Faster R-CNN) [86] and You Only Look Once (YOLO) series. Faster R-CNN is a two-stage object detection model that was developed based on R-CNN [87] and Fast R-CNN [88]. YOLO series is a one-stage object detection network that has both a high detection speed and good accuracy. In this section, some CNN-based methods proposed by researchers for defect detection in PCBs are presented and discussed.

A. CNN-BASED METHODS FOR SURFACE DEFECTS
Ding et al. [89] proposed a tiny defect detection network for PCBs named tiny defect detection network (TDD-net) based on Faster R-CNN. As the PCB images acquired by industrial camera always have a high resolution and the defect regions always occupy a very small portion compared with the whole image, the traditional Faster R-CNN, which produces anchors using 3 scales and 3 different ratios, is not suitable for tiny defect detection. Thus, inspired by YOLOv2 [90], the author applied k-means clustering to the PCB training dataset bounding boxes to seek reasonable anchor scales automatically. Then, data augmentation methods, including adding Gaussian noise, changing light, image rotating, flipping, random cropping and shifting, were implemented. To maintain the features that have low semantic level, the architecture of Feature Pyramid Network (FPN) [91] was adopted. In FPN, the features with low resolution and strong semantics are connected with features that possess high resolution and weak semantics by lateral connections from top to down, and it predicts features on each level of the pyramid network. To mitigate the impact on the detection accuracy caused by the small and imbalance dataset, online hard example mining [92] was applied in the whole training phase to improve the quality of RoI proposals.
In the work [89], 693 images with 6 different defects, namely, missing hole, mouse bite, open circuit, short, spur and spurious copper were synthesized. One image contained several defects of the same type. After training and testing, TDD-net achieved a better performance with 98.9% mean average precision (mAP) than other state-of-the-art methods, including Faster R-CNN with different backbones and FPN. This work achieved excellent detection mAP benefited by the fusion of multiscale features using FPN. However, this algorithm is a two-stage frame and has huge parameters, meaning that the detection speed is obviously slower than that of one-stage network. The use of the full connection layers took up a large amount of parameters. RoI pooling was still used in this work, resulting in the loss of translation invariance of subsequent network features, which would affect the final positioning accuracy. Moreover, the whole retained RoI proposals by the RoI pooling layer will go through the full connection layer and be calculated separately instead of shared calculation between them, leading to huge repetitive computations. All of these shortcomings makes this method not able to achieve real-time detection in real PCB industry.
Adibhatla et al. [93] used the YOLO model to detect defects in PCBs. In order to achieve fast and accurate detection simultaneously, tiny YOLOv2 [94] network modified from YOLOv1 [95], was introduced in this method. The backbone of YOLOv2 is a Darknet-19 network with 19 convolution layers and 5 MaxPooling layers, which is smaller than traditional Visual Geometry Group (VGG) [96] network with comparable accuracy. The floating-point computation of Darknet-19 is reduced to about 1/5 of VGG-16 to allow for faster arithmetic. In tiny YOLOv2, the Darknet network is clipped to less convolutional layers to get faster inference speed. Besides, batch normalization layers are added behind each convolutional layers to make the model converge rapidly. For the model in [93], 11000 images with 11 defect types were collected for training. The detection accuracy of 98.82% for PCB cosmetic defects was accomplished. This work just used tiny YOLOv2 to execute PCB defects detection without modification or innovation for the algorithm. In addition, all the defect types were labelled as a single type of defect, which means that the regions were labelled as either not good or good, reducing the detection difficulty. More performance metrics such as F-score or mAP were missing in this work, and the good detection accuracy only was achieved by small local images cropped from whole PCB images, which make it not very convincing for real PCB inspection application.
Hu and Wang [16] proposed a PCB surface defect detection method based on improved Faster R-CNN and FPN. Faster R-CNN with the backbone of Residual Neural Network (ResNet50) [97] was adopted for the input image as the detector. FPN was used to fuse deep and shallow features, which is considered a powerful way to promote the detection accuracy of small objects. Shuffle V2 [98] residual units were implemented to decrease the computation of the whole network. The Guided Anchoring Region Proposal Network (GARPN) algorithm was used to produce more accurate anchors, and then RoI pooling layers were executed to obtain proposals. Lastly, the fully connected layers were used to classify and perform bounding box regression to accomplish the final defect detection.
Data augmentation was used in [16] to obtain a total of 12000 defect images. This method achieved a detection mAP of 94.2% which was better than that of other state-ofthe-arts including Faster R-CNN [86], RetinaNet [99] and YOLOv3 [100]. Although the introduction of Shuffle network reduced the computation cost, but the running time of this method reached 0.078 seconds per image which was longer than YOLOv3 with the backbone of MobileNet [101]. The reason is that this model was still a two-stage detection model based on Faster R-CNN. Furthermore, this model struggled to detect open defects due to the open defect in the training set appeared in diverse characteristics. This model accomplished faster detection than original Faster R-CNN by some useful tricks, but it did not reach the real-time detection yet. VOLUME 11, 2023 Liao et al. [102] used YOLOv4 [103] with an improved backbone network to detect surface defects in PCBs. In this model, Darknet53, which has higher memory consumption in YOLOv4, was replaced by the lightweight network MobileNetV3 [104] to speed up the detection. The MobileNetv3 uses depth-wise separable convolution to capture feature maps and includes two steps: depth-wise convolution and pointwise convolution. The number of parameters in depth-wise separable convolution is much less than that of normal convolution, making the whole parameters is about 40% less than original YOLOv4.
The model proposed in [102] produced the highest detection accuracy of 98.64% mAP than that of Faster R-CNN [86], RetinaNet [99], Single Shot MultiBox Detector (SSD) [105], YOLOv3 [100] and YOLOv4 [103]. Significantly, the inference speed was satisfactory at 56.98 frames per second by RTX3080 GPU. However, the numbers of each defect type were maintained manually to be approximately equal so that the dataset was balanced in samples, which is contributed to the good performance. Imbalanced samples from real PCB industry usually cut down the detection metrics of common CNN models. Besides, each image in the dataset contained only one defect. Thus, the ability of the method to detect various defects in one image was not proved. Furthermore, the real-time inference speed was achieved for cropped images with small size of 416 × 416 pixels, instead of a whole PCB images. The size of a whole PCB image in this work is about 70 times larger than cropped one. Hence, the detection speed of an entire image must be brought down sharply.
Adibhatla et al. [106] applied YOLOv5 [107] to detect PCB defects because of its good structure, high speed and smaller size compared with previous YOLO versions. In this experiment, three different size YOLOv5 models (small, medium and large) were used. Mosaic data augmentation, splicing four images by random scaling, random cropping and random arrangement, was implemented in this method to rich the dataset and decrease the load of GPU with a small batch size. The backbone of this model was improved by inserted the Cross Stage Partial Network (CSPNet) [108] into Darknet53, making it lightweight while maintaining accuracy. Additionally, Path Aggregation Network (PANet) [109] was combined with FPN to fuse various level features, which should improve the detection precision, especially for small size targets.
In the work [106], all the three different size YOLOv5 models achieved high detection precision. The large YOLO-v5 model had the best detection accuracy of 99.74%. Nevertheless, this work just applied YOLOv5 models to carry out PCB defects detection task without modification or innovation for the algorithm. Only defective or non-defective targets were distinguished. The exact types of defects were not classified and the defect confidence scores were not high. Moreover, similar to previous works, in order to reduce the computation cost, this article only detected defects from cropped images with small size of 400 × 400 pixels, instead of a whole PCB images. Hence, inspection engineers must seek a balance point between the detection precision and speed when the model is applied for a whole PCB image with high resolution.
Zhang et al. [110] modified the Fully Convolutional One-Stage (FCOS) [111] network to achieve surface defect detection in PCBs. The authors replaced the backbone network of ResNet101 [97] with a light network of MobileNetV2 [112] to decrease model parameters. The convolutional block attention module [113] was applied in MobileNetv2 to promote the feature extraction performance. PANet-based FPN was implemented instead of traditional FPN in the model of FCOS. Besides, the bounding box regression loss function was replaced with distance-intersection over union (IoU) loss to boost the detection accuracy for smaller defects.
This model proposed in [110] achieved a detection mAP of 44.3%, which was much higher than that of the traditional FCOS model. The detector achieves proposal free and anchor free, significantly reducing the number of parameters. In addition, by eliminating the anchor frame, the detector completely avoids complex IoU calculations and matching between the anchor boxes and the ground truth boxes during training, which reduces the total training memory footprint by about two times. Thus, the speed was increased by 30% to 37.6 milliseconds inference time compared with that of Faster R-CNN in this work. But, only 1455 images based on the dataset in [89] were collected for training by this model. As a result, this model had a bad mAP of 44.3% for high resolution image, which was 5% lower compared with Faster R-CNN due to the abandon of anchors and the lack of data.
In the real PCB industry, the defect class distribution is always imbalanced. Few previous deep learning models concentrated on this situation. Zhang et al. [114] proposed a cost-sensitive residual convolutional neural network (CS-ResNet) to overcome the problem of imbalanced data. In the study, a cost-sensitive adjustment layer was applied to the fully connected layer of ResNet [97]. After the softmax function was applied, a weighted loss function was obtained based on the imbalanced degree of the training dataset. The weight w t was calculated by Equation (2), where α is an adjustment factor, which is a constant greater than or equal to 1.0, cnt r is the number of real defects, cnt p is the number of pseudo defects. Then, the weight w t was involved in the calculation of softmax cross-entropy loss function, which means that the imbalanced degree of the dataset was considered part of the loss function.
The introduction of weight w t in [114] mitigated the effects of imbalanced dataset. The proposed CS-ResNet achieved a better performance than standard ResNet and other CNNbased methods. The sensitivity of CS-ResNet was 0.89, which is a 12% increase over that of ResNet. One of the drawbacks of this method is that a suitable adjustment factor α needs to be chosen for different datasets with various imbalanced degrees. It performs binary classification successfully, which is not guaranteed for multi-type defects detection. For real PCB inspection tasks, there will be different kinds of defects. The parameter w t based on total real defects and total pseudo defects cannot represent the imbalanced degrees of each type, making it hard to get a better performance.

B. CNN-BASED METHODS FOR SOLDER JOINT AND COMPONENT DEFECTS
To overcome the issue that supervised learning algorithms always need thousands of annotated images, Dai et al. [115] proposed a novel method based on active learning and semi-supervised learning for solder defect detection. YOLOv3 was used to localise the solder joints. The pretrained VGG-16 model was used as the features extractor. Principal component analysis was implemented to reduce the dimensions of extracted features. Then, an SVM classifier was applied to the labelled data. K-means clustering was implemented to analyse the clustering structure of the whole images. Through a combination of these two diverse learning rules, a section of the unlabelled images can be annotated both automatically and manually. Then, the classifier was executed again based on the augmented labelled dataset until the stop criteria, resulting in a good classification performance finally.
In work [115], the YOLOv3 model achieved solder joints localisation with a mAP of 95.1%. Then, during the classification phase, a partition of unlabelled samples were annotated automatically with a small error rate of less than 1.5%. Overall, this model achieved a detection precision of 95.16% and 87.68% for insufficient and shifting solder datasets, respectively. However, this method decomposed the detection task into localisation and classification respectively, which consumed more time than one completed CNN-based model. The localisation speed was about 0.34 seconds per image with 1024×1024 pixels, which was not fast enough. The classification phase by SVM and clustering also was not efficient. Besides, this method concentrated on only two defect types of solder joints: insufficient solder and shifting solder, which is not appropriate to real-world solder joint inspection tasks. Furthermore, the number of selected clusters was a tricky problem in this method, making it not generalized to other datasets perfectly.
Li et al. [116] proposed a deep ensemble model combining hybrid YOLOv2 and Faster R-CNN to achieve PCB solder defect detection. The authors developed the retraining process, which was able to utilise the exception data, such as false positive and false negative data, to train the previous network and produce a renewed model with better performance. After several cycles, the model can decrease the false alarm rate significantly to 20%-30%. The rule was that only the defects detected by both Faster R-CNN and hybrid YOLOv2 are accepted as defects and highlighted. This was achieved by setting the IoU between two predicted bounding boxes larger than the IoU threshold.
The ensemble method proposed in [116] achieved a 97.45% detection rate and a 10-15 seconds inference time for a high resolution image successfully. When this model was applied to another production line, it still obtained good performance after seven retraining processes. However, the retraining cycles must cost huge time and computation resource. The inference speed was also considerably slow, which was far from real-time detection. Besides, during the training process, engineers had to confirm, judge and label the exception data to obtain the retraining data, which means that this method is non-automatic and artificially costly.
Sezer and Altan [117] combined the deep learning network with image matching to detect solder paste defects. Image matching was used to identify the qualified solder joints and unqualified solder joints. Meanwhile, the solder joint pads were extracted by image matching to build the dataset for CNN classifier. For unqualified solder joint pads, containing uncorrectable solder, missing solder, excess solder, short circuit and undefined object, CNN model with the rectified linear unit activation function and AdaMax [118] optimisation method was applied to distinguish them. Common data augmentations such as flipping, rotating, shifting, blurring image were applied expand the dataset.
After image matching, the CNN model achieved 96.4% classification accuracy in 100 epochs, representing good performance. This method was designed for special PCB defects and a direct integration of image matching and CNN classifier. The detection performance highly depended on the output from image matching. In addition, templates were essential for image matching, which is tedious and demanding to be prepared.
Shen et al. [119] proposed a lightweight detection network for PCB components classification (LD-PCB) and a lightweight PCB component character recognition network (CR-PCB). In LD-PCB network, the stem block and dense block from PeleeNet [120] were used to extract features. Inspired by Hu et al. [121], the context-aware RoI pooling method was applied to improve the detection accuracy for small objects. In CR-PCB network, spatial transformer network was applied to reduce the deformation of the raw images, feature sequence extraction layer was utilised to extract features of characters, recurrent neural network layer was implemented to predict the label distribution and attention mechanism converted label distribution into the final recognition result. Based on previous outputs, the information bank that contains component names and characters was established. Then, a defect analysis module counted the information of PCB components and executed template matching or image comparison was implemented to identify defects, such as missing insertion, incorrect insertion, wrong text and missing text.
In the method proposed in [119], the LD-PCB achieved a detection mAP of 85.8% within 0.036 seconds per image. The CR-PCB achieved a recognition mAP of 96.7% within 0.018 seconds per image. The whole system achieved a defect detection accuracy of 95%. This detection strategy was a three-stage algorithm containing component classification, text recognition and image comparison successively. More VOLUME 11, 2023  stages made the method performing a high detection accuracy, but increased the processing time considerably. The whole running time must exceed the range of real-time detection as the combination of LD-PCB and CR-PCB had already cost 54 milliseconds per image. The template matching or image comparison also was time-consuming.
The common approaches used by previous works are CNN networks such as Faster R-CNN and the YOLO series. CNN models are able to automatically extract image features and simplify image pre-processing so that the detection accuracy and speed can be promoted effectively. Furthermore, compared with traditional image processing methods and machine learning methods, CNN-based methods are robust to the environment and noise due to their multi-level feature extraction ability. Moreover, in most cases, reference images can be discarded in deep learning models. Deep learning-based defect detection works are tabulated in Table 5. However, these CNN-based methods still need to be improved. The main limitations are summarized as follows: • Most of them without template matching or image comparison cannot accomplish real-time detection and produce high accuracy at the same time, especially for high resolution PCB images. The main reason is CNNs should contain more layers so that they are deep enough to extract deep features for precise detection. However, a deeper model means many parameters and heavy computations, leading to much more inference time. A balanced point needs to be achieved or some targets should be sacrificed to attain high detection accuracy or speed. For example, methods of [16] and [89] based on Faster R-CNN achieved good performances in mAP, but they are far away from being able to perform real-time detection, which is important in real industries. References [93], [102], and [106] based on the YOLO series obtained real-time detection with excellent detection accuracy, but the datasets they used are designed specially, or they only achieved easy tasks such as binary classification instead of real detection and localisation.
• CNN-based models always need high volume dataset to obtain good detection performance, especially good generalized ability. But PCB defects are difficult to be acquired as most factories must adopt various measures to ensure quality control. The PCB defect types are also diverse, meaning that a dataset containing all types of defects is impossible to be constructed. Moreover, labelling the dataset needs experienced expert knowledge and can expend great effort and time.
• For CNN models, a huge volume dataset means longer training time ranging from several hours to dozens of hours using a general PCB dataset. Thus, some methods utilized powerful GPUs to decrease the training time significantly. However, powerful GPUs are costly and is unusual in common PCB inspection equipment.
• Some CNN-based methods only achieved binary classification of defective and non-defective. The best one achieved 6 types of defects detection, which is not enough for real PCB inspection applications.
• CNN-based methods always have large amount of parameters, which are too weight to be deployed in real PCB inspection applications. Hence, the size of CNN models need to be decreased without losing detection accuracy. CNN models have achieved PCB defects detection with stunning results successfully. In the real applications, the most important issue is how to compress and deploy them in the embedded devices used for inspection. The commonly used methods for model compression and acceleration can be roughly divided into four categories: parameter pruning and quantization, low-rank factorization, transferred/compact convolutional filters and knowledge distillation [122], [123]. Parameter pruning and quantization mainly reduce redundant parameters in the model. Low-rank factorization uses tensor factorization to estimate the parameters of neural networks. The transfer/compression convolution filters are designed with a special structure, which can reduce the parameter space and save memory. Knowledge distillation involves training a larger model and then a smaller model to achieve the same results as the larger model. These four methods are independently designed, but they complement each other. In practical applications, they can often be used together to further compress or accelerate the model for PCB defects detection.

VII. PERFORMANCE COMPARISON AMONG EXISTING METHODS
To make the performance comparison among existing CNNbased methods, a new PCB dataset was built based on 10 high resolution PCB images from the public HRIPCB dataset [89]. Similarly, various defects are produced in each image by Photoshop to build a new dataset containing 800 PCB images with an average resolution of 2,777 × 2,138 pixels. Then, the original high resolution images are cropped into 1981 sub-images with the size of 640 × 640 pixels. Compared with [89], the difference is that at least 2 types of defects exist in each image of our dataset. There are maximum 6 types of defects in each image, including missing hole, mouse bite, open, short, spur and spurious copper. Data augmentation techniques such as flipping, rotation etc. are applied to this small dataset. Finally, our training set with 11093 images and validation set with 2774 images are built, respectively. Using our PCB defect dataset, we compare the detection performances of existing state-of-the-art models, including Faster R-CNN, RetinaNet, SSD, YOLOv3 and YOLOv5. The comparison results are reported in Table 6. The experiments are run on a general computer with one NVIDIA GeForce GTX 1660 Super GPU. All the detection results are acquired at the input image resolution of 2240 × 2240 pixels.
According to Table 6, Faster R-CNN with backbone of ResNet101, as a representative of the two-stage object detection models, achieves a mAP of 90.6%. RetinaNet with backbone of ResNet101 performs a mAP of 96.2%, which is 5.6% more than Faster R-CNN. However, the detection speed of RetinaNet has no marked increase. SSD-Lite has the least parameters, resulting in the significant sacrifice in precision. YOLOv3-tiny obtains the highest mAP of 99.4% and fastest speed of 19 frames per second, demonstrating the desirable superiority in PCB defects detection. But all of them do not reach the real-time detection speed, and models are still too weight to be applied in real PCB inspection industry.

VIII. RESEARCH CHALLENGES A. TRENDS IN PCB DEFECT DETECTION
As stated earlier in this paper, PCB defect detection methods can be mainly divided into four strategies. The first detection strategy is manual visual inspection, which allows experienced experts to capture flaws and types of defects. However, with its low efficiency, high cost and poor robustness, it had become increasingly impractical and is thus being eliminated. After the industrialised AOI machine was developed, many works started to improve the inspection performance of methods based on traditional image processing and machine learning algorithms. More than 60 works were conducted on these two strategies from 1980s to the present because of the ongoing evolution of classical machine learning algorithms. Even recently, some researchers continue to propose novel ideas to modify existing algorithms to enhance the detection accuracy and speed. The last PCB defect detection strategy is deep learning. More than 30 reviews were conducted on this strategy between 2012 and 2022. In the past several years, with the explosion of the use of CNNs for object detection, an increasing number of researchers have become interested in using various modified CNN models for PCB defect detection and other surface inspection of industrial products. The trend of using CNNs to detect defects will continue for a long time due to the rapid development of CNN models and the computation power. VOLUME 11, 2023 B. RESEARCH CHALLENGES Traditional image processing-based methods and machine learning-based methods are not fully automatic and real time. Thus, as mentioned in Section VIII-A, deep learning-based PCB defect detection method is an important research focus in the future. Deep learning-based methods completely depend on the computer for automatic detection, which has the obvious advantages of fast detection speed, high recognition accuracy and robust adaptability. Some researchers have applied deep learning methods to obtain acceptable results compared with traditional methods in PCB defect detection, but studies on deep learning methods are still not abundant. The previous review and introduction indicate that several challenges exist for current deep learning methods used for defect detection in PCBs.

1) OPEN PCB DEFECT DATASET
For the supervised deep learning method, few open datasets of PCB defects are available, especially for component defects and solder joint defects. The existing open datasets mainly focus on component detection or cosmetic defect detection with only one type of defect in one image. Having a large amount of images directly contributes to the performance of CNNs. Thus, having numerous defect images is necessary for the training of CNN models. Also, the process of data acquisition and labelling is time consuming.

2) DETECTION WITH BOTH HIGH ACCURACY AND SPEED
For surface defect detection, without subtraction, most deep learning methods classify inputs into two classes only: defective with location identification and non-defective. The specific defect types are not recognised. In other cases, some works are able to detect defect location and type identification, but the dataset was designed specially, which means that one image contained only one defect or several defects belonging to one type. Even though a few models such as TDD-net have achieved defect location and type identification, these models should balance the detection mAP or accuracy and speed. Thus, the difficulty lies in achieving high detection mAP and speed simultaneously. In addition, the number of types is limited to about six, which also needs to be increased.

3) TINY COMPONENT DEFECT DETECTION
With the development of micro-electronic technology, the assembled PCB is becoming increasingly smaller. Hundreds of tiny components can be mounted on a small board, which is difficult for human eyes to inspect. Component defects may include missing and misaligned components. Some researchers have focused on the detection of component types. However, few studies focus on component defect detection only depending on the deep learning method without image subtraction or matching. One of the possible reasons is that collecting PCBA component defects is more difficult than collecting cosmetic defects. An assembled PCB always has a mass of parts such as resistors, capacitors and chips, thus being a more complicated detection environment for CNN models. Image matching is always applied to reduce the inference time.

4) SOLDER JOINT DEFECT DETECTION
For deep learning-based solder joint defect detection, current methods always rely on outside assistance, such as image matching or registration and manual confirmation or inspection during the detection process, which means that they are not fully automatic. Without assistance, they can only detect the qualified and defective classification with good performances, thus indicating that the detection classes are limited to a small range, which does not coincide with the real industrial manufacturing process.

IX. FURTHER RESEARCH DIRECTIONS A. OPEN PCB DEFECT DATASET
Having a large number of defect images is indispensable for the training of deep learning algorithms. The acknowledged public synthetic PCB cosmetic defect dataset was published by [89] from Peking University. However, its drawback is that one image contains only one type of defects. Further work, such as using Photoshop, can be performed to enrich this dataset so that one image contains at least two types of defects. In addition, some public datasets of solder joint defects and components defects can be downloaded. After they are modified, labels can be marked by software such as LabelMe [124] or LableImg [125]. During the collection process, data augmentation should be executed to obtain high-volume datasets that include different kinds of PCB and PCBA defects.

B. DETECTION WITH BOTH HIGH ACCURACY AND SPEED
Previous work indicated that the YOLO series algorithms can achieve precise and real-time detection, thus making them applicable in real industrial manufacturing. The advanced YOLOv5, YOLOx [126] or YOLOv7 [127] can be used to detect several types of defects in one image, and the performance can be verified through a comparison with other state-of-the-art models. The limitations should be analysed and discussed. An improved model focusing on tiny object detection and model compression can be proposed to achieve better PCB defect detection performance.

C. TINY COMPONENT DEFECT DETECTION
As discussed previously, few deep learning models have been used to detect PCB component defects. Deep learning models can be applied to detect missing or misaligned components. Instead of having a simple context environment for cosmetic defects in PCB, component defects are usually surrounded by many chips, resistors and capacitors. Thus, the deep learning algorithm used for detecting component defects should pay more attention to the context environment around targets. Some tiny object detection models can also be introduced.

D. SOLDER JOINT DEFECT DETECTION
For solder joint defect detection, current deep learning models depend on image matching or registration and manual confirmation during the detection process. These methods can only achieve binary classification without additional operations. An end-to-end CNN model or Transformers [128] based model can be proposed to localise and classify solder defects at the same time. Compared with components, solder joint defects are smaller. Thus, the combination of global context ideas and multi-scale feature learning methods can be used to improve the performance. Moreover, because mostly tiny targets are the focus of PCB defect detection, the depth of the model backbone should contain more spatial information instead of semantic information.

E. PCB AND PCBA DEFECT DETECTION
Finally, on the basis of previous ideas that have become reality, a CNN model that is able to detect bare PCB cosmetic defects and PCBA solder joint and component defects simultaneously with acceptable performance can be proposed. This approach will be much more challenging than implementing only one of them, which means that the model must be robust to different environments and backgrounds. Such a model can greatly benefit the PCB-related manufacturing industry.

X. CONCLUSION
Quality control in the PCB manufacturing process is usually a critical problem. Various defects, including surface, component and solder joint defects, inevitably appear due to mishandling or technical faults. In this paper, different defect detection methods based on conventional image processing techniques and deep learning models are reviewed and compared. Traditional image processing-based defect detection methods achieve acceptable detection accuracy, but they are time consuming and sensitive to the environment and the inference image. With its multi-level feature extraction and automatic learning abilities, CNN-based defect detection algorithms have overcome such issues and achieved significant performances in both accuracy and speed. However, deep learning-based methods still has some limitations, such as the collection of datasets, binary classification and small range of defects. On the basis of such shortcomings, some suggestions are proposed and discussed, which are believed to achieve good results in the future.