Real-Time Prediction of Welding Penetration Mode and Depth Based on Visual Characteristics of Weld Pool in GMAW Process

The penetration depth of welding seam can reflect welding quality fundamentally, during the gas metal arc welding (GMAW) process, the penetration depth of welding seam fluctuates over time. At present, it lacks of reliable sensing method to predict penetration depth fluctuation accurately in real time. To solve the above problem, in this paper, proposing a real-time prediction method for weld penetration mode and depth based on two-dimensional visual characteristics of weld pool, establishing a monocular vision sensing system, extracting the area, length and width of weld pool as key two-dimensional visual characteristics. Taking the extracted current frame characteristics of weld pool as the input, the weld penetration of welding seam corresponding to current frame as the output, based on support vector machine (SVM) and back propagation (BP) neural network respectively, the real-time prediction models for weld penetration mode and depth were established. The predicted results show that the established models can accurately predict the penetration mode and penetration depth of welding seam in real time.


I. INTRODUCTION
In the whole welding process, the penetration depth that the soldering seam can reach is an important parameter concerned by scholars, therefore, real-time prediction and control of the weld penetration depth accurately have always been a research hotspot in the welding field [1]- [3]. A lot of previous researches focused on the detection of the fluctuation of weld penetration mode under different welding parameters.
In terms of weld penetration mode detection, visual sensing was the most widely used detection method. Wu et al. [4] designed a set of portable vision sensor in variable polarity plasma arc welding process, acquired multiple keyhole images on the back of workpiece under different welding conditions, extracted the area and tilt angle of keyhole as the characteristic parameters, used the extracted The associate editor coordinating the review of this manuscript and approving it for publication was Chi-Tsun Cheng . keyhole characteristics to predict weld penetration mode. Jiajia et al. [5] used near infrared vision sensing method to acquire multiple frame molten pool images under different penetration conditions in Aluminum alloy double wire welding process. The feature parameters such as the area, perimeter, half length, width and parabolic coefficient of the molten pool were obtained by extracting the contour of the molten pool. Based on neural network, a model for identifying the penetration mode of aluminum alloy double wire welding was established. Chen et al. [6] proposed a method to predict the penetration mode of gas tungsten arc welding (GTAW) process based on data-driven approach, used computer vision method to extract the key features of the molten pool surface, conducted experiments under various welding conditions to establish the database, used two supervised machine learning methods such as linear regression to test the database, finally, the importance of the surface characteristics of molten pool was analyzed. Liu et al. [7]- [9] used 3D vision sensing system to extract the features of molten pool, researched on the method for predicting and controlling the backside bead width, and controlling 3D molten pool surface in GTAW Process.
In terms of weld penetration depth detection, during the welding process, the visual expression of weld penetration depth is the deepest of the lower surface of molten pool, therefore, many domestic and overseas scholars directly detected the depth of weld pool through various detection methods. According to the different sensing technology, it can be divided into: weld pool vibration method [10], ultrasonic sensing method [11], infrared sensing method [12], X-ray method [13], etc. However, these detection methods have corresponding shortcomings, for instance, there is a certain delay in the weld pool vibration method, so it is hard to achieve real-time detection. Ultrasonic, infrared and X-ray detection methods have the shortcomings such as complex structure and high cost, which can not be widely used in the actual welding production. Some scholars predicted the weld penetration depth by means of numerical simulation, Wu et al. [14] established a three-dimensional molten pool surface sensing system, used image processing, 3D reconstruction methods to extract the key features of molten pool surface, based on the single ellipsoid heat source, established an analytical model of welding heat process, used the surface features of molten pool to modify and optimize the analytical model and measured the weld penetration depth in real time.
However, during the gas metal arc welding (GMAW) process, the penetration depth of soldering seam fluctuates over time. In this paper, a real-time prediction method for weld penetration based on two-dimensional visual characteristics of weld pool was proposed. Support vector machine (SVM) is a kind of generalized linear classifier which classifies data in binary way by supervised learning, back propagation (BP) neural network is a kind of multilayer feedforward neural network trained by back propagation algorithm, based on SVM and BP neural network respectively, the real-time prediction models of weld penetration mode and depth were established, the experimental results verified the feasibility of the prediction model, the designed prediction model can be used for real-time detection of weld penetration in GMAW process, which lays the foundations for on-line control of weld penetration, so as to ensure the welding quality.

A. PENETRATION MODE PREDICTION EXPERIMENT IN GMAW PROCESS
Conducting experiments in the cold metal transfer (CMT) welding process, steel plate was selected as the welding base metal, the thickness of steel plate was 2 mm, the welding wire material was stainless steel, the diameter is 1.2 mm. The vision sensing system of weld pool was composed of color charge coupled device (CCD), field programmable gate array (FPGA) and computer, as shown in Figure 1, the color CCD was mounted on the welding torch by fixture, moved with welding torch, FPGA sent out fixed frequency signal to control color CCD weld pool image acquisition.

B. PENETRATION DEPTH PREDICTION EXPERIMENT IN GMAW PROCESS
Conducting experiments in the CMT welding process, steel plate was selected as the welding base metal, the thickness of steel plate was 5 mm, the welding wire material was stainless steel, the diameter is 1.2 mm. The whole experimental system included the vision sensing system and laser positioning system, as shown in Figure 2, the vision sensing system of weld pool was also composed of color CCD, FPGA and computer, the color CCD was used to acquire weld pool image.
In addition, we designed a laser positioning system, the laser positioning system contains laser, monochrome CCD and computer. The central wavelength of the laser was 450nm, the laser was fixed on the welding torch, and the laser irradiated the upper edge of the welding wire, the monochrome CCD was mounted on the welding platform to capture the position of laser points. During the welding process, FPGA sent out synchronization signal to trigger color CCD and monochrome CCD, according to the position of laser points in each frame acquired monochrome image, we can accurately match each frame acquired color image with the actual position of welding seam.

C. EXPERIMENTAL SCHEME
In the experiment of GMAW welding penetration mode prediction, the welding current has an important influence on the weld penetration mode, in this paper, the welding current was set in the range of 70 A to100 A, the step size was 10 A, and the welding speed was 3 mm/s, the specific welding parameters are given in Table 1. During the welding process, FPGA sent out a signal to trigger the color CCD acquire weld pool image, the frequency of signal is 1000 Hz, we extracted the area, length and width of the weld pool as the key characteristic parameters, provided training and test data for weld penetration mode prediction model. Taking the extracted current frame characteristics of weld pool as the input of prediction model, the penetration mode of soldering seam corresponding to current frame as the output of prediction model.
In the experiment of GMAW welding penetration depth prediction, the welding current was set in the range of 140 A to 150 A, the step size was 10 A, and the welding speed was 5 mm/s, the specific welding parameters are given in Table 2. During the welding process, FPGA sent out synchronization signal of 1000 Hz to trigger color CCD and monochrome CCD, we extracted the area, length and width of weld pool as the characteristic parameters. Taking the extracted current frame characteristics of weld pool as the input of prediction model, the penetration depth of soldering seam corresponding to current frame as the output of prediction model.

III. EXTRACTION OF VISUAL CHARACTERISTIC PARAMETERS OF WELD POOL A. DEFINITION OF TWO-DIMENSIONAL CHARACTERISTICS OF WELD POOL
The weld pool image was acquired by color CCD, extracting the weld pool contour subsequently, calculating the area, length and width of the weld pool. Figure 3 is the sketch of the weld pool image, in the Figure, the X axis is the row coordinate axis, the Y axis is the column coordinate axis, and the X axis is consistent with the welding direction.

1) WELD POOL AREA
The weld pool area is defined as the sum of all pixel points in the weld pool image contour. The specific calculation equation is as follows: where RowL i denotes the row coordinate value of the pixel point on the left edge of row i when scanning the row of weld pool contour one by one, RowR i denotes the row coordinate value of the pixel point on the right edge of row i, N is the number of rows of the weld pool contour.

2) WELD POOL LENGTH
The weld pool length is defined as the number of pixel points contained between two edge points of the weld pool contour with the largest distance in the X-axis direction. The specific calculation equation is as follows: where Row max denotes the row coordinate value of the front edge point of weld pool, Row min denotes the row coordinate value of the rear edge point of weld pool.

3) WELD POOL WIDTH
The weld pool width is defined as the number of pixel points contained between two edge points of the weld pool contour with the largest distance in the Y-axis direction. The specific calculation equation is as follows: where Column max denotes the column coordinate value of the top edge point of weld pool, Column min denotes the column coordinate value of the lowest edge point of weld pool. Figure 4 is the waveform of welding current in CMT process, and Figure 5 shows the whole acquired weld pool image in a CMT cycle. The cycle of CMT process is about 14 ms, as shown in Figure 5, in the peak phase of CMT process, the weld pool image is strongly affected by arc light, in the base phase of CMT process, the weld pool image is almost not affected by arc light, and the signal-to-noise ratio is very high. Therefore, in the experiments of weld penetration prediction, we selected the first frame image in each CMT cycle, which acquired in the base phase, extracted the two-dimensional key characteristics of the weld pool.

B. EXTRACTION OF TWO-DIMENSIONAL CHARACTERISTICS OF WELD POOL
According to the acquired colored image of weld pool, firstly, extracting the contour of weld pool by binary processing, then filling the cavities in the weld pool contour, as shown in Figure 6, it is the processed result for a certain frame weld pool image, finally, calculating the two-dimensional characteristic parameters of weld pool.

IV. ESTABLISHMENT OF PENETRATION MODE AND DEPTH PREDICTION MODEL FOR GMAW WELDING PROCESS A. ESTABLISHMENT OF PENETRATION MODE PREDICTION MODEL FOR GMAW WELDING PROCESS
Conducting the welding experiment by using the data in Table 1, the length of each welding seam was set to 8 cm, the experimental results are shown in Figures 7, 8 and 9, at the bottom, it is weld pool images corresponding to different positions of welding seam. In the case of welding current is 100 A, when the moving length of the welding torch is less than 7 mm, the welding seam is not penetrated, and when the moving length of the welding torch exceeds 7 mm, the welding seam begins to penetrate. In the case of welding current is 90 A, when the moving length of the welding torch is less than 14 mm, the welding seam is not penetrated, and when the moving length of the welding torch exceeds 14 mm, VOLUME 8, 2020   the welding seam begins to penetrate. In the case of welding current is 70 A, in the whole welding process, the welding seam is in the mode of non-penetration.
Color CCD acquired weld pool image at the frequency of 1000 Hz, we selected the first frame colored weld pool image in each CMT cycle, which acquired in the base phase,   extracted the area, length and width of weld pool as the characteristic parameters. Taking the extracted current frame characteristics of weld pool as the input, the weld penetration mode of welding seam corresponding to current frame as the output, by using SVM, the real-time prediction model of welding penetration mode was established. In the case of the welding current is 90 A, the welding seam includes two modes: non-penetration and full-penetration, used the data acquired at the welding current of 90 A as training set, and the data acquired at the welding current of 100 A and 70 A as test set, verified the prediction accuracy of the established model.
There are 1965 samples in the training set and 3987 samples in the test set. Before training, we need to preprocess the data set, the data pretreatment process is also known as normalization, all data in the data set are mapped into [0,1], and the specific calculating formula is given as: where x origin and x new denote the data before and after normalization respectively, x max and x min denote the highest and lowest values in the original data set respectively.

B. ESTABLISHMENT OF PENETRATION DEPTH PREDICTION MODEL FOR GMAW WELDING PROCESS
Conducting the welding experiment by using the data in Table 2, the length of each welding seam was also set to 8 cm, after completion of welding, cutting the welding seam evenly into two parts along the welding direction, selecting half of them for grinding, corrosion, then the variation of VOLUME 8, 2020 welding penetration depth along the welding direction can be observed by using a body stereomicroscope. As shown in Figure 10(a), it is the metallographic diagram of a certain section of welding seam observed by the body stereomicroscope in the case of welding current is 110 A, Figure 10(b) shows the extracted welding seam outline, according to the purple baseline of the lower surface of base metal and scale in Figure 10(a), calculating and determining the baseline of the upper surface of base metal in the image, the distance between the baseline and the lower surface contour of welding seam is the penetration depth, so we can calculate the actual penetration depth of welding seam. Then by using the laser positioning system in the experiment, we can match each frame acquired color image with the actual position of welding seam. Selecting the first frame colored weld pool image in each CMT cycle, which acquired in the base phase, extracting the characteristic parameters such as area, length and width of the weld pool. Taking the extracted current frame characteristics of weld pool as the input, the weld penetration depth of welding seam corresponding to current frame as the output, based on BP neural network, the real-time prediction model of weld penetration depth was established, the neural network included two hidden layers, according to the test effect of neural network by setting the different number of neurons in hidden layers, the number of neurons was set to 6 and 3 respectively, the model structure is shown in Figure 11. Each group of welding parameters welded two welding seams, using the data acquired in the first welding seam as the training set of the model, and the data acquired in the second welding seam as the test set of the model. Through the computer, the CPU is i7-8700K, we trained the prediction network model in MATLAB software, and finally converted the prediction model to the C++ language based on Microsoft Visual Studio environment.
Under the welding current of 140 A, there are 884 samples in the training set and 904 samples in the test set. Under the welding current of 150 A, there are 822 samples in the training set and 857 samples in the test set. Before training, all data in the data set need to be normalized according to equation (4).

A. PREDICTION RESULTS AND ANALYSIS OF WELDING PENETRATION MODE FOR GMAW PROCESS
In the case of welding current is 90 A, the critical point of penetration mode of welding seam corresponds to the first frame image acquired in the base phase of the 304th CMT cycle. In the case of welding current is 100 A, the critical  point of penetration mode of welding seam corresponds to the first frame image acquired in the base phase of the 144th CMT cycle. Using 0 represents non-penetration mode, 1 represents full-penetration mode. Using the data acquired at the welding current of 90 A as training set, training the prediction model, the predicted results of weld penetration mode under the welding current of 100 A and 70 A are shown in Figure 12: The prediction accuracy of the model is given in Table 3, it can be found that the prediction model of weld penetration mode for GMAW process can effectively predict the weld penetration mode in real time.    Table 2, each group of welding parameters welded two welding seams, using the data acquired in the first welding seam as the training set of the model, and the data acquired in the second welding seam as the test set of the model. The predicted results of weld penetration depth under two different welding currents are shown in the Figure 13.
It can be found that during the welding process, the weld penetration depth fluctuates over time, but on the whole, it tends to increase. And the weld penetration depth will increase as the welding current increases.
The prediction accuracy of the established model can be evaluated by mean absolute error (MAE), the calculation equation of the MAE is: where d is the predicted penetration depth, d is the actual penetration depth, and m is the total number of data groups in the test data set.
The prediction MAE of the model is given in Table 4, it can be seen that the prediction model of weld penetration depth for GMAW process can effectively predict the weld penetration depth in real time.

VI. CONCLUSION
1) Based on the color CCD, a weld pool vision sensing system was designed, which can real-time extract the twodimensional characteristics of weld pool in the whole welding process.
2) Based on SVM and BP neural network respectively, the real-time prediction models for weld penetration mode and depth were established. The experimental results show that the predicted accuracy of weld penetration mode is higher than 95.7%, the predicted MAE of weld penetration depth is less than 0.122 mm.
3) The future work includes extracting more key visual characteristics of weld pool, further improving the prediction accuracy of weld penetration depth prediction model, and researching online control method for weld penetration depth.