Feature Extraction of Laser Welding Pool Image and Application in Welding Quality Identification

Most of the existing laser welding quality identification methods are post-weld identification or low-speed identification (Welding speed below 120 m/min). Efficiently online monitoring of laser welding can take the advantages of laser welding for high-speed and deep-penetration welding. How to eliminate interference information (such as metal vapor, plasma splash, etc.) in the laser welding process, accurately and quickly extract the feature information of welding quality evaluation, and identify defects is a major problem that laser welding online monitoring technology needs to solve urgently. In this paper, the optimized dark channel prior anti-interference processing algorithm can remove the interference of image. The feature information extraction algorithm based on contour and OTSU threshold segmentation are used to extract the features of the welding image that collected by the image acquisition system. Then, the image is classified as a specific defect by the trained BP neural network classification algorithm. Experiments with 304 stainless steels have proved that this method can effectively remove the interference of metal vapor and plasma splash on the feature information, and achieves 97.18% accuracy rate of the binary classification test and 91.29% accuracy rate of the six-classification test. The processing time of the entire algorithm is about 0.3 ms and it can meet the real-time requirements of high-speed laser welding.


I. INTRODUCTION
As a new type of welding technology, laser welding is one of the most ideal technologies to realize high-speed and deeppenetration welding. Because of its high energy density and ability to weld heterogeneous materials [1], it is developing rapidly in the field of industrial processing and modern intelligent manufacturing [2]- [4]. However, due to various process parameters, defects inevitably occur in laser welding.
The current non-destructive testing methods for the quality of laser welding welds are mainly post-weld testing and lowspeed online monitoring. It is mainly divided into welding internal defect detection methods (such as: radiation [5]- [9], ultrasonic [10]- [14], etc.), welding surface defect detection methods (such as: visual [15], [16], structured light [17]- [21], etc.) and on-line monitoring methods during low-speed welding (such as: short shutdown [22], near-infrared camera [23]). Post-weld identification also has disadvantages such as long The associate editor coordinating the review of this manuscript and approving it for publication was Senthil Kumar . identification times, low efficiency and poor traceability. In some cases, complete identification is not possible and only sampling identification can be performed. Miao et al. proposed a weld quality recognition method based on CNN (Convolutional Neural Network) that has the advantage of high accuracy [24], and the time for single identification is 3.2 s, but this 3.2 s processing time is too long for synchronous detection. Indimath et al. [22] proposed a welding defect detection method based on ultrasonic that requires a short shutdown of the production equipment for detection. Dorsch et al. proposed a weld feature detection method based on the near-infrared camera [23] that is difficult to achieve ultra-high frame rate detection and complete identification in high-speed laser welding (120 m/min and above).
In order to solve the many shortcomings of the abovementioned detection methods, it is of great significance to develop online monitoring technology of laser welding quality that can simultaneously monitor the quality of weld during the welding process. Therefore, monitoring the welding process through machine vision and image processing VOLUME 9, 2021 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ technology, and simultaneously extracting important information for evaluating welding quality during the welding process, is the hot direction for solving the above-mentioned problems. However, when images are acquired continuously, the keyhole boundaries are heavily disturbed by bright light. Therefore, many scholars in the world have studied various imaging methods and image processing methods [4], [25]- [30]. Katayama et al. [30] studied the reflected laser beam and radiated light as monitoring signals during laser welding of aluminum alloys to elucidate the correlation between monitoring signals and welding phenomena during the formation of weld defects. Wen and Gao used a high-speed camera to acquire images of the plume generated during laser welding in the UV and visible wavelengths, and then classified the samples by using the feature parameters of the splash images and the k-nearest neighbor classification method. As the result, the recognition rate is higher with the average gray level and entropy of the image as the feature parameters [31]. Zhang et al. [32] used 660 nm and 850 nm filters to combine the images collected by the dual cameras to obtain a clear image of the weld pool and predict the width of the weld. Chen et al. [33] used a coaxial CCD camera installed on the welding head to shoot the molten pool, and studied the relationship between the area of the molten pool and the laser power, defocus distance, welding speed and other process parameters through various image processing algorithms.
Meanwhile, many different defect detection methods based on texture analysis have been proposed [34]. Fekri-Ershad and Tajeripour [35] proposed a surface texture defect detection method based on single dimensional local binary patterns. It has high detection rate and low computational complexity. Timm and Barth [36] performed Weibull features in texture defect detection. It can detect local deviations of texture images in an unsupervised manner with high accuracy, and be applied in real-time applications. Susan and Sharma [37] achieved an automated high accuracy texture defect detection method. This method used non-extensive entropy with Gaussian gain as the regularity index and computed locally from texture patches through a sliding window approach. In the end, an automatic defect detection with no manual intervention was achieved.
On the other hand, machine learning algorithms such as neural networks are also widely used in the evaluation and classification of welding defects. Shevchik et al. [38] proposed an improved adaptive kernel algorithm. By using Gaussian mixture and complex features, the classification accuracy of this algorithm is between 85.9% -99.9%. Chen et al. obtained the plasma radiation information during welding by the optical fiber probe, classified the defects of optical fiber laser welding [39] by using the plasma spectral data and neural network algorithm, and compared the performance of SVM (Support Vector Machine) and neural network. Zhang et al. [40] built a multi-sensor laser welding monitoring system and used CNN (Convolutional Neural Networks) to classify three welding defects. The robustness of BP neural network was compared with that of CNN. Shevchik et al. [41] used hard X-ray radiography to obtain laser welding images. And a variety of CNN models are used to classify defects in X-ray images. The confidence of the defect classification ranges between 71% and 99%, and a computation time per classification task as low as 2 ms.
However, above approaches has their own shortcomings in different aspects. It has either too much time spending or too less accuracy. In addition, the device requirement of those approaches is much.
Due to the widespread use of laser welding in various industries, a more effective online quality identification method is needed to identify defects.
At present, ultrasonic, structural light and X-ray equipment are commonly used in laser welding quality inspection equipment, but the equipment is complex and the cost is high. In our study, only an ordinary camera is needed to monitor the welding quality, which greatly reduces the cost. However, camera imaging is easily disturbed by metal vapor and plasma splash, so feature information is lost seriously, and only contour information can be extracted. Using HU moment and BP neural network to identify defects can not only extract features well, but also meet the requirements of high-speed processing.
In this paper, the optimized dark channel prior antiinterference processing algorithm and the feature information extraction algorithm based on contour and OTSU threshold segmentation are used to extract the features of the welding image that collected by the image acquisition system. Then, the image is classified as a specific defect by the trained BP neural network classification algorithm. Experiments that using 304 stainless steels have proved that this method can effectively remove the interference of metal vapor and plasma splash on the feature information, and achieves 97.18% accuracy rate of the binary classification test and 91.29% accuracy rate of the six-classification test. The processing time of the entire algorithm is about 0.3 ms and it can meet the realtime requirements of high-speed laser welding. In addition, the device has a simple structure and can monitor the quality of weld simultaneously during the welding process.

II. LASER WELDING QUALITY FEATURE EXTRACTION METHOD
During the laser welding process, the molten pool-keyhole movement contains a wealth of welding quality information, and the morphology feature of the molten pool-keyhole directly reflect the welding quality, weld morphology, welding defects and other conditions [42]. A high-speed camera can be used to obtain a clear image of the molten pool, and image processing technology is used to extract relevant feature information, such as the contour of the molten pool, the area of the molten pool, etc. Then statistical analysis methods are proper way to establish the relationship between the feature of the molten pool and welding defects [43]. In this way, the quality of laser welding can be judged.
In order to solve the existing problems, a method of morphology feature extraction for laser welding based on machine vision is proposed in this paper. This method uses anti-interference algorithms with different parameters for processing when extracting different features. And the features can be extracted synchronously in the welding process and make full use of the fast laser welding speed.
In order to accurately extract the feature information of weld state, the following three steps are used in this method: (1) anti-interference pre-processing based on dark channel prior; (2) extraction of keyhole feature information based on contour characteristics; (3) extraction of weld feature information based on OTSU threshold segmentation. They are respectively used to exclude the interference information in the weld image, extract the keyhole feature information and the feature information of the weld width.

A. ANTI-INTERFERENCE PROCESSING BASED ON DARK CHANNEL PRIOR
According to a paper published by He in 2009, a haze removal algorithm based on dark channel prior was proposed [44], and the dark channel is defined as: where J dark (x) represents the dark channel value of the pixel x; J c represents each channel of the color image; (x) represents a window centered on the pixel x. According to statistical experience, a haze-free image often has the conclusions as: The prior statistical conditions in He's paper were obtained from 5,000 natural haze-free images. For the laser welding conditions described in this paper, the optimized dark channel prior algorithm is also effective for metal vapor and plasma splash, since the effects of metal vapor and plasma corona on the image are similar to those caused by haze in nature.
The atmospheric scattering model as: where I (x) represents haze image; J (x) represents haze-free image; A represents global atmospheric light intensity; t (x) is the transmittance, that is, the part that is not scattered when the light enters the imaging device through the air. The I (x) means the light intensity which received by sensors. It is made up of two parts. One of it comes from object reflection. The J (x) t (x) part of the equation represents that the light reflects from the object surface and it scatters before the sensor receives. Another one comes from light source. The A (1 − t (x)) part of the equation represents that the light is emitted from the source and it also scatters before the sensor receives. The global atmospheric light intensity is hard to get from images, because it's light intensity when the light emitted from the source. In the algorithm, the atmospheric light intensity is estimated.
According to the atmospheric scattering model, only need to obtain the values of t (x) and A to calculate the hazefree image J (x). Since the welding image does not have a clear distinction between the front and back scenes, solving the global atmospheric light intensity A does not achieve good results when using the algorithm of OTSU thresholding of natural scenes and solving the global atmospheric light intensity in segments [45]. In our anti-interference algorithm, simply take the RGB pixel corresponding to one thousandth of the brightest pixel of the dark channel image, calculate the average value of the corresponding channel, and take this as the global atmospheric light intensity A, then good antiinterference effect can be obtained.
Global atmospheric light intensity formula as: where A c represents atmospheric light intensity of the channel c; Sum c represents sum of the values of channel c in the RGB pixels corresponding to the thousandth brightest pixel point of the dark channel image; N represents one thousandth of the total number of pixels in the original image. Transmittance formula as: where I c represents a haze image of channel c; ω represents retention coefficient of haze.
In the haze removal process of images with natural haze, the requirements for the solved transmittance t (x) are relatively loose. Image areas with different depth of field also have certain light scattering in the absence of haze. For making the image more reality after haze removal, the retention coefficient ω of haze is usually 0.95. However, under laser welding conditions, there is almost no effect of depth of field, and the reflection of the metal plate around the weld is easily interpreted as part of the haze by the algorithm.
If ω = 1 is applied to the image, the edge of the weld is not clearly maintained. The test results are that when ω = 1 (complete haze-free) extracting the keyhole feature information and when ω = 0.9 (maintaining high contrast at the weld edge) extracting the weld feature information is a better option. To improve the computational speed, the gradientoriented filtering method [46] is used in this paper to solve the transmittance.
In the process of solving the dark channel of natural scenery images, the commonly used (x) range radius is 5-25 pixels. According to the characteristics of laser welding images, the smaller (x) range can better remove the interference caused by metal vapor and plasma splash to the image. The edge of the feature will also be clearer.

B. KEYHOLE FEATURE INFORMATION EXTRACTION
When defects occur in the laser welding process, the morphology of the keyhole changes more obviously, and it is feasible to use the contour of the keyhole as the feature VOLUME 9, 2021 of welding defects. The contour of the keyhole in different welding states are shown in Fig. 1.
For a pure welding image, the contour of the keyhole can be extracted simply through the OpenCV API.
Since the contour image of the keyhole is relatively simple, the contour geometric moment [47] can be used to distinguish it, and it can meet the requirements of calculation speed and accuracy at the same time.
The n-order geometric moment of the contour as: where M i,j represents the n-order geometric moment of the contour; I (x, y) represents gray value at point (x, y).

C. WELD FEATURE INFORMATION EXTRACTION
Extract the welding seam image with a width of 70 pixels behind the keyhole position, then perform OTSU threshold segmentation [48] and morphological filtering processing on it. The comparison between original image and processed image is shown in Fig. 2.
For the image shown in Fig. 2(b), the following method is used to locate the weld position.
Take a 5 (width) * 70 (height) subset of the image and scan from left to right. When the number of white points in the image is greater than 60%, take the column number of the center column of the subset and record it as C 1 . For another edge of weld, take a subset of the same size and scan from right to left. When the number of white points in the image is greater than 60%, take the column number of the center column of the subset and record it as C 2 . The weld width can be obtained by the formula as: where C 1 , C 2 represents column number of the image corresponding to the center of the left and right weld edges; std represents calibration of camera, the unit is mm/pixel; W represents weld width. The 7 Hu moment values and the feature value of the weld width are taken as the features extracted from the image, and the classification algorithm of machine learning take those values as input to evaluate the quality of the weld.

III. IMAGE FEATURES AND WELD QUALITY EVALUATION
Traditional machine learning classification algorithms (such as SVM [49], etc.) still lack support for multi-classification problems, meanwhile the neural network has higher fitting ability and accuracy, and has better robustness [50].
In this section, 7 HU moment values and weld width were used as inputs to construct BP neural network as shown in Fig. 3.
A BP neural network with 100 neurons in a single hidden layer is constructed, and 8 feature values calculated based on the image are used as input to identify defects in the image. The 6 outputs correspond to the Sound well weld state, Lack of fusion state, Burn through state, Dislocation state, Large gap state and Inclusion state in the welding quality respectively.
The BP neural network constructed in this section only processes 8 feature values instead of directly processing the image, which greatly reduces the amount of calculation and reduces unnecessary information interference, which helps to achieve high-speed image processing capabilities.

IV. EXPERIMENTAL SYSTEM AND DATASET A. EXPERIMENTAL SYSTEM
The experimental system includes: a laser welding system with shielding gas and an imaging acquisition system with auxiliary light. Fig. 4(a) shows the general schematic diagram, and Fig. 4(b) shows our experimental equipment.
The laser system used a fiber laser with a wavelength of 1064±10 nm. The laser beam was focused on the working surface through a defocus lens with a spot diameter of 0.4 mm. Welding parameters are shown in Table 1. This set of parameters was adopted in all experimental data to ensure the consistency of feature extraction and training standards, as well as the quality of welding. During the welding process, the base material was clamped to a rotating device weighing several hundred kilograms. Due to the huge mass of the transmission device, it took some time to accelerate or slow down the welding speed to reach uniform speed.
In this paper, high-power laser welding experiments were conducted on 2 mm thick, 1420 • C melting point 304 stainless steel, which is widely used in industry and has significant defect features.
The HD image acquisition system used Optronis CP70-12-M-167 camera with Active Silicon AS-FBD-4XCXP6-2PE8 image acquisition card. The system had 4000 FPS, 480pix * 480pix RGB image acquisition capabilities. The monitoring parameters are shown in Table 2.
In the monitoring of the welding process, the high-speed camera directly obtained the RGB image of welding. For imaging, metal vapor and plasma splash were high intensity interference light sources, so it was difficult to achieve ideal results by directly observing holes. Our attempts have shown that the use of lasers with a central wavelength of about 1064 nm and austenitic stainless steel, a narrow-band filter of about 808 nm and auxiliary illumination laser of the same wavelength could well suppress interference from plasma and metal splash, and the interference of metal vapor could be removed more than 95% by using an anti-interference algorithm based on dark channel prior. The whole experimental device was placed on the robotic hand, which allows us to conveniently load and accurately locate the welding position during the debugging process.
A Windows computer equipped with AMD Ryzen 5900X CPU, 64G RAM, and GTX1650 GPU was used in the image processing system. In this paper, the GenICam Transport-Layer API, OpenCV API and the C++ programing language were respectively used to collect images, extract the image features and run the BP neural network algorithm.

B. DATASET
In this paper, three kinds of welding defects which are closely related to the shape of keyhole and two kinds of welding defects which are very sensitive to the weld width were analyzed, namely, the Dislocation, the Large gap, the inclusion, the lack of fusion and the burn through. The first three could be collectively referred to as defects generated by the assembly, the last two for penetration defects. The actual weld image is shown in Fig. 5.
Due to the large mass of the transmission drive, a stable welding speed couldn't be achieved immediately when welding was first started, and this part of the image data needed to be discarded. Same thing at the end. In order to obtain enough data for training and testing, experiment used the parameters in Table 1 (1450 W for the lack of fusion state; 2150 W for the burn through state, and the rest of states refer to Table 1). A total of 7129 pieces 512pix * 512pix image of welding image data of 304 stainless steel were collected under different welding conditions. It was divided into six-classifications by experienced welding engineers, namely sound well weld state (1000 samples), lack of fusion state     Table 3, 4, 5 and 6. Dataset A corresponds to the binary classification test of whether there were defects. Datasets B and C correspond to the sensitivity between feature information and each defect of assembly defects and penetration defects, respectively. In Dataset D, all defects were identified together to test the model's ability to identify all defects. All the data were subsets of the same complete dataset.

C. DATA PROCESSING
The flowchart of entire process is shown in Fig.6. The flowchart of entire process is shown in Fig.6. In the data processing part, the images in the dataset were first processed by the anti-interference algorithm, then the images were respectively processed by the keyhole feature information extraction algorithm and the weld feature information extraction algorithm, and finally the extracted feature values (7 Hu moment values and 1 weld width value) were used for the training of BP neural network.

V. EXPERIMENTAL RESULTS
In this paper, Dataset A is used to verify the influence of antiinterference algorithm on the accuracy of feature extraction. Datasets B and C is used to verify the sensitivity of the classification method to assembly defects and penetration defects, respectively. Dataset D is used to verify the identification effect of all six classification defects.

A. INFLUENCE OF ANTI-INTERFERENCE ALGORITHM ON DEFECT IDENTIFICATION
Metal vapor and plasma splash have obvious interference to the feature extraction of welding images. After processing with anti-interference algorithm, the interference of metal vapor and plasma splash in the image is significantly reduced,   and the edges of keyhole and weld remains high gradient, which can be effectively identified by the feature extraction algorithm. Comparison of anti-interference algorithm before and after processing is shown in Fig. 7.
The core of the anti-interference algorithm used in this paper is the haze removal algorithm. And traditional indicators for evaluating haze removal algorithms usually include PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) [51]. Since the essence of welding image processing is to remove interference information, and after processing, a large amount of texture information that is not related to feature extraction is removed from the image. The evaluation indicators of traditional haze removal algorithms are difficult to evaluate the effect of anti-interference algorithms proposed in this paper. Therefore, the classification results of the neural network and traditional machine learning algorithm are used in this paper to evaluate the effect of antiinterference processing. Traditional machine learning algorithm (SVM () [49], KNN (K-Nearest-Neighbors) [52], Naive Bayes Model [53]) and single hidden layer BP neural network are used to conduct binary classification test on Dataset A, and the results are shown in Table 7. All models use the same inputs and outputs (8 feature value which is 7 Hu moment and 1 weld width as inputs and 2 class which is with and without defects as outputs).
All comparison method were trained in MATLAB R2020a. In the SVM model, we choose the parameters as kernel: Gaussian kernel which is also called Radial Basis Function (RBF) kernel, Kernel scale: 2.6, Box constraint level: 1, Multiclass method: one vs one, Standardize data: true. In the KNN model, we choose the parameters as Number of neighbors: 100, Distance metric: Euclidean, Distance weight: Equal, Standardize data: true. In the Naïve Bayes Model, we choose the parameters as Distribution name for numeric predictors: Kernel, Distribution name for categorical predictors: MVMN, Kernel type: Gaussian, Support: Unbounded.
As shown in Table 7, the accuracy of all method was increased and the maximum accuracy improvement was 6.9% in SVM model. Obviously, after the anti-interference algorithm, the weld state features could be extracted more accurately and more representative of the image.

B. IDENTIFICATION RESULTS OF WELD DEFECTS
To ensure that the classification algorithm has sufficient generalization capability, this paper uses a 5-fold crossvalidation [54] approach for training. Fig. 9 -12 shows the BP neural network confusion matrix of Dataset A, Dataset B, Dataset C, Dataset D, respectively. The numbers in the green squares represent the number of samples for which the predicted value of the neural network is the same as the target value, i.e., the number of correctly classified samples; the numbers in the red squares represent the number of samples for which the predicted value of the neural network is different from the target value, i.e., the number of incorrectly classified samples.
It can be seen from Fig. 9 that for the binary classification test, the trained BP neural network model can obtain a classification accuracy of 98.37%. The misjudgment is due to the fact that the features of some defect images are similar to those of sound well weld images, mainly lack of fusion images, which can also be seen in Fig. 11. The main feature difference between the lack of fusion images and the sound well weld images is only in weld width, while the feature weight of single feature information is limited in the process of neural network calculation, which is easy to cause the VOLUME 9, 2021   misjudgment of defects. It can be seen from Fig. 10 and 12 that the classification accuracy of defects with Large gap is the highest in both the assembly defect classification test and the last six-classification test. This is because the Large gap defects have significant changes in the shape of the keyhole and the width of weld, and are significantly different from other defects. It can be seen from Fig. 10 and 12 that various types of defects are easily classified as Dislocation defect. This is because in the image of Dislocation defect, the weld width does not change significantly, and the features of  keyhole fluctuate significantly. Some image of Dislocation defect has some similarities with all kinds of defects in the shape of the keyhole and the width of the weld. In the final six-classification test, the classification accuracy of various types of welding images ranged from 82.20% to 98.43%, and the overall classification accuracy was 91.29%. And the time of entire procedure is about 0.3 ms which include system I/O spending, anti-interference algorithm spending, feature extraction algorithm spending and BP neural network spending in ours experiment computer.
As for the multi-classification results of the comparison method and our method, they are shown in the Table 8.
As shown above, our method has much higher accuracy. That is because for SVM model, it was designed with poor support for multiple classification problems, because it was designed for binary classification problems. For KNN model, its support for multi-classification problems is well, but it is difficult to classify complex data because KNN processes data in original dimension, not higher dimension. For naive bayes model, due to its assumption of sample independence, the classification effect is not good when the sample attributes are correlated with each other. In our data, the samples are related to each other. This gives it the lowest accuracy in binary classification.
In conclusion, the anti-interference algorithm can significantly improve the classification accuracy of the various algorithm. In the BP neural network algorithm, the classification accuracy can be improved by 5.1%. Since the dataset is continuously sampled from the welding video, it can reflect the real welding conditions well and has good generalization ability. Due to the combined effect of the keyhole features and the weld width feature, the BP neural network algorithm has the highest classification accuracy for Large gap defect, which can reach 98.43% in the most demanding sixclassification test. The processing time of the entire algorithm is about 0.3 ms and it can meet the real-time requirements of high-speed laser welding. And the proposed algorithm only requires a visual-light camera and a computer.

C. TIME COMPLEXITY ANALYSIS OF ALGORITHM
The image size in our dataset is 512pix * 512pix. When image size increases, time cost mainly increases in the feature extraction part. The time complexity of feature extraction part is O n 2 and the other part of our algorithm is O (n). In general, our algorithm's time complexity is O n 2 .

VI. CONCLUSION
In the high-power laser welding process, metal vapor and plasma splash can greatly interfere with the feature information of weld quality. Therefore, it is difficult to extract feature information for weld quality evaluation, and establish a direct correspondence between weld image features and weld quality. To solve the above problems, this paper uses an anti-interference algorithm based on dark channel prior to process the image, and uses the trained BP neural network model to identify the defect by using the keyhole features and weld width feature. Experiments have proved that the algorithm can effectively remove the interference of metal vapor and plasma splash on the welding quality feature information, and can accurately identify defects in the feature information of the welding image. It can achieve an overall classification accuracy of 97.18% in the binary classification test, and 91.29% in the six-classification test. The processing time of the entire algorithm is about 0.3 ms and it can meet the real-time requirements of high-speed laser welding. And the proposed algorithm only requires a visual-light camera and a computer. However, the classification accuracy of the model will be affected by imaging. And for different kinds of materials, the configuration of the imaging system needs to be redesigned and the model needs to be retrained. In summary, our approach needs to be redesigned or retrained for different materials or different imaging system. PENG XU was born in Anqing, Anhui, China, in 1999.
From 2018 to 2020, he was doing research in physical direction with a Professor in the laboratory, including the thermal lens effect of the medium in the light field and its imaging law. He is currently with the Department of Applied Physics, Donghua University.