A Multi-Sensor Data Fusion System for Laser Welding Process Monitoring

Most existing laser welding process monitoring (LWPM) technologies focus on detecting post-process defects. However, in sheet metal laser welding applications such as welding of electronic consumer products during mass production, in-process defect detection is more important. In this article, a compact LWPM system using multi-sensor data fusion to detect in-process defects has been built. This system can collect the time series of plasma intensity, light intensity and temperature data for feature analysis. To verify the system’s effectiveness, a plasma-light-temperature dataset has been compiled, which consists of 5,836 samples of nine classes, including one positive class and eight negative classes of typical in-process defects. A multi-sensor data fusion network based on a convolution neural network for in-process defect detection, called IDDNet, has also been proposed. Experimental results have demonstrated that IDDNet can achieve better multi-classification results than the support vector machine, with an overall accuracy of 97.57%. In particular, considering this monitoring process as a binary classification problem, IDDNet can achieve a 99.42% accuracy. Moreover, IDDNet can reach an average speed of 0.79ms per sample on a single GTX 1080ti graphics card, which meets the real-time requirement for industrial production. The proposed LWPM system has been successfully verified in real applications of sheet metal laser welding.


I. INTRODUCTION
Laser welding is widely used in electronic consumer products, automobiles and other industries due to the advantages of contactless processing, high precision and high speed [1]- [3]. The quality of laser welding is easily affected by in-process defects such as poor process parameters, surface impurities of materials, operation errors [4]. In-process defects may cause post-process defects such as pores and lack of fusion. For example, defocusing distance is one of the main process parameters during laser welding.
The associate editor coordinating the review of this manuscript and approving it for publication was Zhongyi Guo .
Incorrect defocusing distance often leads to pores and lack of fusion [5]. These defects could potentially affect the safety performance of the welded components [6].
Most existing laser welding process monitoring (LWPM) technologies focus on detecting post-process defects [7]- [9]. In automated production lines, post-process defects usually appear continuously if in-process defects are not troubleshot in time. For example, due to the negligence of suppliers, sometimes a batch of incoming materials have impurities or wear on the surfaces, which will probably cause a batch of substandard products. This means a huge loss if only post-process defects are concerned, especially in the manufacturing process of high-end electronic VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ consumer products. Moreover, most existing post-process defect detection technologies focus on large-size workpieces [7]. For small electronic consumer products, manual sampling and destructive detection methods are still dominant, which are inefficient and hard to guarantee the product quality. Hence, in automated mass production such as laser welding of electronic consumer products, detecting in-process defects is more important.
In practice, it is challenging to detect in-process defects as the manufacturing process involves many tools, technologies and parameters. For example, to check whether the defocusing distance is incorrect, a direct method is to measure it through a set of specific instruments and methods. Gao et al. [10] measured defocusing distance in real-time through a YAG laser, a position sensitive detector, many convex lenses and a set of geometric methods. However, these instruments and methods cannot detect surface impurities of materials. In other words, different in-process defect detection would require different instruments and methods. If all kinds of in-process defects are required to detect, due to the heavy workload and high complexity, it is impossible to integrate all the corresponding instruments and methods into a single system. Therefore, a compact system with strong universality is preferred.
To develop a compact system, a common practice is to build the relationship between the process signals during laser welding and the in-process defects. However, it is still difficult to establish an explicit and direct relationship due to the complex laser welding process [7]. The challenge is twofold: 1) to find out the appropriate process signals that have a relationship with certain in-process defects, even if the relationship is implicit; and, 2) to extract and fuse the useful features of these process signals for detecting the corresponding defective samples. In literature, various forms of process signals can be captured by different sensors in the laser welding process [11]. There are mainly four types of sensors: visual [12], acoustic [13], optical [14] and thermal [15]. Visual sensors are often used in LWPM systems to assess the laser welding process's quality by monitoring melt pools or keyholes [16]- [18]. Usually, a high-resolution image or video is required to ensure the accuracy of the visual system. Since a large amount of image data needs to be captured and processed, it is difficult to achieve real-time performance for in-process defect detection in rapid production lines [3]. Furthermore, the hard light in the laser welding process causes saturation in the images, which will reduce the system's accuracy. Due to background noises in production lines, acoustic sensors are not suitable to be integrated into LWPM systems for detecting in-process defects [7]. As a result, this research focuses on optical and thermal sensors.
The optical and thermal process signals in the laser welding process have been studied for years. Santhanakrishnan et al. [19] found that the molten pool's temperature is related to the process parameters such as laser power, laser scanning speed and overlapped laser spot. You et al. [20] investigated that the keyhole formation has a significant influence on the laser's reflecting light intensity. Knag [21] used the time series of plasma intensity and temperature data to assess the laser welding process's quality through a simple statistical method. However, with only plasma intensity and temperature information, it is difficult to identify the types of in-process defects. Therefore, a compact LWPM system with multiple sensors has been developed to capture and analyze the time series of plasma intensity, light intensity and temperature data during laser welding. Because of the extreme complexity of the laser-material interactions during laser welding, it is elusive to establish an explicit physical model that links the above signals and in-process defects. As machine learning (ML) can provide an effective data-driven approach to correlate inputs and outputs without knowing too much domain knowledge [22], in this research, in-process defect detection is therefore considered as a multi-classification problem, where multi-sensor data are fused for classification based on ML methods.
Conventional ML methods include decision tree (random forest) [23], support vector machine (SVM) [24], Naive Bayes [25]. In general, these methods heavily rely on heuristic hand-crafted data fusion and feature extraction involving extensive domain knowledge [26]. Without hand-crafted data fusion and feature extraction from the raw data, these methods can only extract low-level features leading to poor accuracy in defect detection and classification [27]. Unlike traditional ML methods, the convolution neural network (CNN) as a deep learning method does not require rich domain knowledge for data fusion and feature extraction [28]- [30]. Instead, CNN can automatically learn how to implement data fusion and high-level feature extraction through a tremendous amount of training data and the stochastic gradient descent algorithm [31]. Hence, CNN is chosen for the proposed multi-sensor data fusion and high-level feature extraction automatically to detect in-process defects.
To summarize, the contributions of this work are as follows: 1) A compact LWPM system with multiple optical and thermal sensors has been developed. This system captures the time series of plasma intensity, light intensity and temperature data simultaneously during laser welding. It then analyzes the features of these signals and identifies the types of in-process defects. To verify the system's effectiveness, a plasma-light-temperature dataset (PLTD) with 5,836 samples has been compiled. These samples contain typical in-process defects.

2) A CNN-based In-Process Defect Detection Network,
called IDDNet, has been proposed to fuse the captured time series and then detect in-process defects.
Experimental results have demonstrated that IDDNet achieved better multi-classification results than SVM, with an overall accuracy of 97.57%. In particular, considering this monitoring process as a binary classification problem, IDDNet can achieve an accuracy of 99.42%. Moreover, IDDNet can reach an average speed of 0.79ms per sample on a single GTX 1080ti graphics card, which meets the real-time requirement for industrial production. The remainder of this article is organized as follows. In Section II, the fore-end signal acquisition part of the proposed LWPM system and the PLTD for typical in-process defects are described. Section III first defines the terms for time series classification and then elaborates on the details of IDDNet. The experimental results are presented in Section IV. Finally, the paper is concluded in Section V with a plan for future works.

II. EXPERIMENTAL SETUP AND DATASET A. EXPERIMENTAL SETUP AND SIGNAL ACQUISITION
The proposed LWPM system has been integrated into a scanning laser welding machine from Han's Laser Technology Industry Group Co., Ltd (Han's Laser), as shown in Fig. 1 (a). The overall schematic structure is presented in Fig. 1 (b).
The scanning laser welding machine mainly includes an SPI pulsed fiber laser, a galvanometer scanner, two vibrating mirrors and a flat field lens. The galvanometer scanner can reflect laser light to the desired position by turning the reflection mirrors to change the laser path.
The fore-end signal acquisition part of the proposed LWPM system includes two different photodiode sensors, a pyrometer sensor and some optical components. The first photodiode sensor is installed behind a 45-degree prism and a band-stop filter to obtain the plasma intensity. After another 45-degree prism, the light intensity will be captured with the second photodiode sensor. Finally, a pyrometer sensor is installed at the end to monitor the temperature data during welding.
In this experiment, the collection of PLTD was conducted on the real production lines of Han's Laser through the proposed LWPM system. The experimental material was Type SUS301 stainless steel with 0.2mm thickness. This type of material is being used in the specific sheet metal laser welding applications of Han's Laser. Furthermore, the laser's output power was 60W, and the welding speed was set to 50mm/s. By manually checking the quality of the welded products in the above applications, these kinds of welding parameters outperform others, with the lowest defect rate. 1) Qualified means no in-process defect occurred.
2) Defocus 2mm refers to the defocusing distance over 2mm. The focus plane above the workpiece is positive defocus, while the focus plane below the workpiece is negative defocus. The defocusing distance of excessively large absolute value leads to the overly low power density acting on the workpiece, making it difficult to reach the purpose of welding. 3) Defocus -2mm represents defocusing distances of less than -2mm. 4) White glue means there is white glue on the surface of the base metal. 5) Missing weld is a widespread operation error. 6) Drift indicates the welding position suddenly drifted. 7) Tilt represents the base metal's tilt during welding, so that defocusing distance was changed. 8) Repetition means to weld again based on the existing welded seam. 9) Water indicates there is water on the surface of the base metal. To emphasize the importance of in-process defect detection, some examples of negative effects caused by typical in-process defects are shown in Fig. 2 (a)-(g).  Fig. 3. As the training of CNN requires as many samples as possible [32], all the samples available were used for experiments.
There are three variates in each sample: plasma intensity, light intensity and temperature. For convenience, each sampling point is used as a unit. The length of each variate is 128, representing a total of 128 sampling points for each variate. Fig. 4 (a)   recognizable distribution interval of each variate. Missing weld (red line) can also be easily differentiated from Qualified because once missing weld occurs, the light intensity will rise sharply while the temperature will drop rapidly. Unlike Defocus 2mm or Missing weld, Water (yellow line) almost coincides with Qualified in most of the time series, but the differences appear at both ends. In summary, each class has its characteristics in these three variates, which shows the possibility of accurate classification to some extent.
T-SNE [33] is a dimensional reduction technology that maps data into two-dimensional space for visualization. One hundred samples from each class were chosen randomly and used T-SNE, as shown in Fig. 5. Apart from White glue, Missing weld and Defocus 2mm, the distributions of other classes are almost overlapped. This well demonstrates the difficulties in the current defect classification problem.

III. METHODOLOGY A. TIME SERIES CLASSIFICATION BASED ON PLASMA-LIGHT-TEMPERATURE DATASET
Based on PLTD, in-process defect detection can be seen as a time series classification problem. Before introducing the proposed approach, the formal definitions for this time series classification are provided as follows.
(i) A univariate time series U can be defined as: where every x i denotes a real value.
(ii) A three-variate time series X can be defined as: where X consists of three univariate time series U , corresponding to plasma intensity, light intensity and temperature, respectively. (iii) PLTD contains 5,836 pairs: 9] (3) where X i denotes a three-variate time series with Y i as the label of the corresponding class. For PLTD, there are nine classes.
The time series classification based on PLTD is to build a classifier that can map from a given three-variate time series X to the predicted probabilities of the nine classes.

B. CONVOLUTION NEURAL NETWORK FOR TIME SERIES
Convolution for time series can be seen as sliding a filter over the time series [32]. An example of convolution is depicted in Fig. 6. The convolution for a centered timestamp t is given in the following equation: (4) where * denotes dot product, and O t denotes the result of a convolution operation applied on an input X of length T with a filter w of length l. b is a bias parameter, and F is a non-linear function such as rectified linear unit (ReLU). After convolution, the following layers are usually pooling and batch normalization (BN) layers [34].

C. ADVANTAGES OF CONVOLUTION NEURAL NETWORK
In recent years, many advanced deep learning methods have been developed, such as the recurrent neural network (RNN), the multi-layer perceptron (MLP) and CNN. RNN is mainly designed to predict output for each timestamp in the time series [35]. Besides, it is difficult to train and parallelize [36]. MLP tends to overfit because it does not exhibit any spatial invariance among a huge number of trainable parameters [32]. Unlike MLP, CNN has the characteristic of weight sharing by using the same filters on all timestamps. Weight sharing contributes to reducing the number of parameters drastically and avoiding overfitting [37]. Therefore, a CNN-based in-process defects network for feature extraction and classification is proposed.

D. IN-PROCESS DEFECT DETECTION NETWORK
The performance of CNN is affected by many factors such as the number of layers, number of filters, kernel size and strides in the convolution layer [37]. In this work, the architecture of CNN proposed by Wang et al. [38] is adopted to design IDDNet, where the first layer has three variates as inputs. The architecture of IDDNet is illustrated in Fig. 7. This architecture first consists of three convolution blocks. Each block contains three operations: a convolution layer followed by a BN layer whose result is fed to a ReLU activation function. The final discriminative layers are comprised of a global average pooling layer [39], a fully connected layer and a softmax layer. Finally, the predicted probabilities of the nine classes are obtained.
All convolutions are designed with strides equal to 1. Moreover, no padding is used. The first convolution contains 128 filters with a filter length equal to 8. The second convolution contains 256 filters with a filter length equal to 5.   At last, the third convolution layer contains 128 filters with a filter length equal to 3.

IV. EXPERIMENTAL RESULTS
This section presents the comparison between the classification results of IDDNet and that of SVM. In addition, to explore each variate's contribution, the experiments on each variate and each combination of two variates were performed separately. All evaluations were conducted with ten times 3-fold cross-validation [40], and then the average value of ten times 3-fold cross-validation was used as the final results.

A. CLASSIFICATION RESULT OF IDDNET
IDDNet was trained under the training parameters with a learning rate of 0.0005 and a mini-batch size of 128. The stochastic gradient descent algorithm with backpropagation was used to minimize the cross-entropy loss function [27] in IDDNet. The training was stopped after 500 training epochs. A confusion matrix [41] was chosen to illustrate the classification results, as shown in Fig. 8. All diagonal elements of the confusion matrix are the maximum values of the corresponding rows. This indicates that most samples could be correctly classified. Fig. 9 shows the confusion probability matrix. Only one sample of Tilt was misclassified into Qualified and Qualified achieved an accuracy of 99.42%. As shown in Fig. 10, the feature map of the last convolution layer of IDDNet was extracted and T-SNE was used for visualization. It can be seen that the distributions of all classes are no longer overlapped and have significant differences, compared with that in Fig. 5.

B. COMPARE WITH CONVENTIONAL METHOD
To illustrate the comparative advantages of IDDNet, it was compared with SVM. Moreover, SVM can solve high-dimensional problems and non-linear problems based on kernel tricks. Fig. 11 shows that IDDNet outperforms SVM. Since SVM heavily relies on heuristic hand-crafted data fusion and feature extraction, it is difficult to find the optimal feature for classification. In contrast, the proposed IDDNet can fuse the high-dimensional data for feature extraction automatically and thus achieve better performances.
The computation time for the above two methods on the same computer (Intel Core i7-6700 HQ CPU and 16.00 GB RAM, GTX 1080ti graphics card) was calculated. Table 1 summarizes the training time and testing time of   In order to explore the contribution of each variate, the performance of IDDNet based on every single variate and each combination of two variates was investigated, as summarized in Fig. 12. The classification result based on temperature outperforms that of the plasma intensity or light intensity, achieving an 89.94% overall accuracy. That means temperature contributes the most to the classification result among the three variates.

V. CONCLUSION AND FUTURE WORK
In this work, a compact laser welding process monitoring system using multi-sensor data fusion to detect in-process defects has been developed. This system first captures the time series of plasma intensity, light intensity, and temperature data simultaneously during laser welding. It then analyzes these signals' features and identifies the types of in-process defects. To verify the system's effectiveness, a plasma-light-temperature dataset (PLTD) has been compiled for experiments, consisting of 5,836 samples. Experimental results have demonstrated that the proposed IDDNet has an overall accuracy of 97.57%, which is a much better multi-classification result than that of SVM. In particular, considering this monitoring process as a binary classification problem, IDDNet can achieve an accuracy of 99.42%. Moreover, IDDNet can reach an average speed of 0.79ms per sample on a single GTX 1080ti graphics card, which can satisfy the real-time requirement for industrial production.
In this research, the proposed system and methods have been verified in the sheet metal laser welding application. In the future, more applications will be studied. Furthermore, the influence of process parameters such as the laser power on the quality of products will be investigated to optimize the process parameters automatically and efficiently. YINGYING CHEN is currently pursuing the B.S. degree in electronic information engineering with Wuyi University, Jiangmen, China. Her research interests include machine learning, computer vision, and signal processing.