Intelligent design of robotic welding process parameters using learning-based methods

With the wide application of multi-layer and multi-pass welding in industry, the traditional manual welding method is difficult to meet the needs of manufacture. Welding Robot has the advantages of stable productivity, ensuring welding quality even in special environment, so the welding robots are used at a growing trend in manufacturing fields to complete different welding tasks. In this paper, an intelligence learning method for welding robot is designed, aiming at the prediction of welding process parameters and bead geometry parameters in the welding process, deep and machine learning algorithms are used for realization. It provides an instruction for the design of process parameters to realize the intellectualization and automation of welding robot. The experimental results show that automatic parameters learning based on machine learning are effective and different learning methods should be selected for different process parameter prediction tasks in order to achieve the best prediction effect.


I. INTRODUCTION
In the field of industrial manufacturing, due to the drawbacks of inefficient welding and poor quality stability, manual arc welding is not suitable for the needs of intelligence. For example, in the task of welding at high buildings, the welding performance of high-strength steel is relatively poor, the welding amount of thick plates is large and weld beads are long, and the welding environment is hard and difficult. Besides, the technical skills of welding workers are at different levels. To a certain extent, these factors affect the stable quality of steel structure of high-rise buildings. Thus, the reliability and safety of engineering quality cannot be guaranteed [1]- [3]. As the basis of intelligent and automatic welding, robot welding has the advantages of improving production efficiency, ensuring welding quality and adapting to intelligent requirements. It has gradually replaced manual welding, met the needs of large-scale production and welding automation, and has been widely used in the field of manufacturing.
However, the welding process is complicated with different factors and many parameters. Different welding tasks demand different requirements and different dynamic working environment. Thus, it is a challenge to realize the intelligent adaptation of welding robot to different tasks, ensuring stable welding quality. In the process of intelligent welding, an important task is to learn and predict the technical parameters of the welding process, for the purpose of accuracy control and need of reusing. Therefore, the comprehensive controlling of process parameters in the welding process plays a vital role in improving the automation welding and ensuring stable welding quality. It is also a part of the comprehensive life cycle assessment of industrial supply chain.
Recently, the key technology to solve the problem of robot welding process control and monitoring is to combine machine learning algorithm or expert system with robot welding process planning. For instance, Kim et al. [4] proposed a new method to predict the technological parameters of machine arc welding, which is composed of neural network and multiple regression. A welding parameter prediction method based on Gaussian process regression and Bayesian optimization algorithm was presented by Dong et al. [5]. Liu et al. [2] proposed a classifier based on adaptive neuro fuzzy inference system, which can be used to evaluate the skill of welder and transfer the intelligence of welder into welding robot. Butdee et al. [6] proposed an oil pipeline machine response model based on fuzzy logic modeling to predict depth penetration. All parameters are defined in fuzzy input and output. Create fuzzy rules for reasoning. The fuzzy output provides a solution for effective decision-making to select optimal parameters, and is directly connected with the welding robot. Cao et al. [7] proposed a neural network integration method composed of radial basis function neural network, BP neural network and generalized regression neural network, and established the correlation among laser pool, keyhole and weld characteristics. Zhou et al. [8] designed a physics based machine learning method based on ensemble learning model to identify the correlation between the deposition stage and its corresponding thermal field. Asif et al. [9] proposed a real-time welding quality monitoring method combined with decision-making strategy to improve welding production efficiency and automation. Aiming at the problem of complex welding physics and time-consuming welding parameterization, Caotu et al. [10] proposed to introduce knowledge-based welding parameterization decision support into robot work unit, combined with advanced collision free off-line programming and advanced sensing technology, developed a flexible welding robot system. DANIEL et al. [11] proposed a framework which combines a methodology for selecting indicators and a multi-objective optimization to improving sustainability pillars, and used this framework to the optimization of a submerged arc welding process.
Deep neural networks have shown impressive power of feature representation in many fields. Compared to shallow architectures, deep neural networks can extract the latent feature representation and better capture the semantic information of data. Recently, deep neural networks were introduced in welding tasks. Cruz et al. [12]introduced a computer vision system by using combined digital image processing and deep learning techniques for welding inspection of liquefied petroleum gas pressure vessels. Bacioiu et al. [13]trained models on an inert gas welding dataset, leveraging the machine learning research, establishing a correlation between the weld pool and surrounding area and the weld quality.
The key of intelligent welding is in using intelligent techniques and machine intelligence to realize automatic welding. If the welding process is an end-to-end system with various inputs and outputs, we can model the process of parameter learning mathematically and use the model as a tool for predicting and optimizing process parameters.
Aiming at the problem of intelligent selection of process parameters in the welding process, this paper explored a variety of learning methods including machine learning, ensemble learning to predict the process parameters in the multi-layer and multi-pass welding process. By comparing the prediction results of various learning algorithms, the appropriate model is selected to realize the intelligence and automation of robot welding, which provides an important instruction for the intelligent control design of welding robot.
The remainder of this article is structured as follows: in Section II, related machine leaning-based works about welding are introduced. Section III is Research methodology, in which we describe BP neural network, CatBoost, XGBoost and CNN in details. Experiments and results are shown in Section IV and Section V concludes our work and provides future application from it.

II. RELATED WORKS
This section mainly describes the application of machine learning algorithm in welding process.
As a learning method of combination optimization, ensemble learning solves a single prediction problem by assembling multiple models to obtain a better combination model. It also allows researchers to design combination schemes for specific machine learning problems to improve the generalization ability of learning.
Xu et al. [14] proposed a weld defect recognition algorithm based on selective ensemble learning to solve the problem of low recognition rate of weld defects in radiographic inspection, which effectively improved the diversity of component learners and the generalization ability of ensemble learners. Adrian et al. [15] constructed an Ensemble Data-Driven Fuzzy Network (EDDFN) for laser welding quality prediction by using the ensemble model and the relevant information provided by the monitoring system.
In the field of welding, there are many ensemble learning algorithms that can be used, and XGBoost [16] is one of the most outstanding algorithm, which was first proposed by Chen et al. in 2016. In their paper, they proposed a novel sparsity-aware and weighted quantile sketch for approximating tree learning algorithm for sparse data, and described a scalable end-to-end tree boosting system XGBoost. Zhang et al. [17] established a model using extreme gradient boosting (XGBoost) machine learning algorithm to predict the temperature distribution of molten pool during direct energy deposition (DED). Based on the XGBoost algorithm, Chen et al. [18] Proposed two data-driven models to identify the penetration state and predict the rib reinforcement. The weld bead width and weld bead reinforcement were measured by line structured light method, and the weld bead penetration was measured by macro metallographic microscope. Qu et al. [19] used machine learning algorithm XGBoost to realize nonlinear correction and cold junction compensation of thermocouple on upper computer, constructed dynamic compensator by particle swarm optimization algorithm to reduce dynamic error, and finally realized on-line monitoring of welding temperature in the form of upper computer software.
Another kind of machine learning neural network is also common in welding field. Chang et al. [20] proposed a BP neural network optimized by Mind Evolutionary Algorithm (MEA) to predict the influence of penetration morphology on weld quality. Zhang et al. [21] used principal component analysis (PCA) to analyze the features of weld pool shadow to reduce its redundancy, and used genetic algorithm (GABP) to improve BP neural network to establish the relationship model between weld appearance and weld pool shadow features. It provides an effective method for real-time prediction of weld morphology and evaluation of welding quality. Geng et al. [22] proposed a BP neural network for welding opti-mization based on genetic algorithm for quality inspection. But one of their common shortcomings is that they do not consider choosing different machine learning algorithms for different parameters. In [23], an image classification algorithm based on convolutional neural network integration was successfully applied to detect the misalignment of metal plates connected by submerged arc welding process.

III. RESEARCH METHODOLOGY
The process parameters of robot welding process include welding process parameters and bead geometry parameters. In this paper, different machine learning algorithms are selected to compare the prediction of these parameters. Therefore, this section focuses on the machine learning models used in the experiment of robot welding process: BP neural network, CatBoost, XGBoost and CNN.

A. BP NEURAL NETWORK
BP neural network is a multi-layer feedforward neural network trained by error reverse propagation algorithm. The error analysis is carried out according to the training results and the expected results, then the weights and thresholds are modified, and the model which can output and predict the results is consistent step by step is obtained. The outstanding advantage of BP neural network is its strong nonlinear mapping ability and flexible network structure. The number of intermediate layers and the number of neurons in each layer can be set arbitrarily according to the specific situation, and their performance varies with the structure. The global error E is shown in Eq.(1): where n is the number of input layer nodes and m is the number of output layer nodes. x j represents the input of the jth neuron. s j is the net input value of the jth neuron, b j is the threshold value, w ij is the corresponding weight, f (·) is the activation function, t p j is the expected output, y p j is the output of the pth sample input to the network, and E p is the error of the pth sample. P is the number of learning samples.
The weights in the BP neural network are obtained by iteration update, which follows the Gradient Descending rules: At iteration step t, the gradients of weights connecting the hidden layer and output layer ∆w ho (t) are updated by Eq. (6) ∆w ho (t) = m j=1 (y p j (t) − t p j (t))y p j (t)(1 − y p j (t))t p j (6) and the gradients of weights connecting the input layer and hidden layer ∆w ih (t) are updated by Eq. (7) ∆w In this study, BP neural network is used for welding parameters learning, and its schematic diagram is shown in Figure 1. There are two different BP neural networks used to predict welding parameters in the robot welding process. Figure 1(a) shows that the welding process parameters are predicted when the welding process parameters are given, and Figure 1(b) shows that the welding process parameters are predicted when the welding process parameters are given.

B. CATBOOST (FOR 'CATEGORICAL BOOSTING')
CatBoost is an implementation of gradient boosting, which uses binary decision tree as base predictors. The gradient boosting decision tree are symmetric trees, which has fewer parameters, and supports categorical variables with has high accuracy. CatBoost is composed of category and boosting, which can process categorical features efficiently and reasonably. In addition, CatBoost also solves the problems of gradient bias and prediction shift, so as to reduce the occurrence of over fitting and improves the accuracy and generalization ability of the algorithm. The advantages of CatBoost are summarized as: it can obtain high model quality without adjusting parameters and it adopts a new gradient promotion mechanism to build the model to reduce over fitting.
When catboost algorithm processes the categorical features in GBDT features, it uses the improved greedy target based statistics to reduce the influence of noise and lowfrequency categorical data on data distribution. The formula is as follows: where p is the added prior term, a is usually the weight coefficient greater than 0, x i k is the classification feature of the kth training sample, Y k ∈ R is a target.

C. XGBOOST(FOR"EXTREME GRADIENT BOOSTING")
XGBoost is an efficient and widely used machine learning algorithm, which is implemented by an improved gradient boosted framework. XGBoost creates a decision tree by adding weak bias-low variance base model (shallow decision tree) in turn. Each tree is constructed to adapt to the residual of previous tree. Every tree in XGBoost can be regarded as a weak base learner, all of these trees are combined to improve VOLUME 4, 2016 (a) Prediction of bead geometry parameters by welding process parameters (b) Prediction of welding process parameters by bead geometry parameters  Figure 2. The objective function of XGBoost in step t is approximately where g i = ∂ŷ(t−1) l y i ,ŷ (t−1) and h i = ∂ 2 y (t−1) l y i ,ŷ (t−1) are first and second order gradient statistic on the loss function. Ω (f t ) = γT + 1 2 λ T j=1 ω 2 j is the regularization term which can reduce overfitting problem, γ and λ are regularization parameters, T is the number of leaves in XGBoost.
XGBoost uses greedy algorithm to traverse all feature partition points of all features. Specifically, The objective function value after splitting is better than the objective function of single child leaf node. Also, a threshold is added to prevent the tree from growing too deep. Only when the gain is greater than the threshold can it be split to form a tree. Each time, based on the previous prediction, take the best to further split to build a tree. The significant advantage of XGBoost is that it can automatically use CPU multithreading for parallel computing, which improve the algorithm to increase the accuracy.

D. CONVOLUTIONAL NEURAL NETWORK
CNN was proposed by Yann Lecun in 1998, it is essentially a multi-layer perceptron. CNN reduces the number of weights of neural networks and makes the network easy to optimize. Also, it reduces the complexity of the model by sharing the common weights, which turns out reduces the risk of overfitting. The structure of convolution neural network includes convolution layer, down sampling layer and full link layer. The flowchart of CNN is shown in Figure 3. Each layer has multiple feature maps, each feature map extracts an input feature through a convolution filter, and each feature map has multiple neurons.
The input of convolutional neural network is where conv2 is the convolution operation function, and the third parameter valid indicates the convolution operation type. W is the convolution kernel matrix, X is the input matrix, and b is the offset.The output is where, φ(X)is the activation function.
The above input-output formula is for each convolution layer. Each convolution layer has a different weight matrix W , and W , X and y are in matrix form. For the last fully connected layer, it is rounded to layer L, the output is y L in vector form, and the expected output is d, there is a total error formula

A. THE DATA
The data used in this experiment are obtained in the actual welding process. The welding process is shown in Figure 4. Figure 4(a) is the picture of the workpiece before welding, Figure 4(b) is the picture of the workpiece during welding, and Figure 4(c) is the picture of the workpiece after welding. Table 1 shows the welding data, including current (I), voltage (U), welding speed (V1), wire feeding speed (V2), weld width (b), weld depth (H) and surplus height (a). The first four are welding process parameters, and the last three are bead geometry parameters.

B. EXPERIMENTAL SETTING
The experimental data are processed by machine learning algorithms: BP neural network, CatBoost, XGBoost and CNN. During the experiment, No.1-28 was used as training data, No.29-41 as test data to predict welding process parameters. Firstly, the BP neural network with two hidden layers is used to train the data. There are 32 nodes in the first layer and 16 nodes in the second layer. The bead geometry parameters are predicted through the welding process parameters. Secondly, in the welding process parameters predicted by the bead geometry parameters, the hidden layer node is 32*16. The

C. RESULTS AND DISCUSSION
There are many parameters that affect weld formation. For different types of welding (spot welding, argon arc welding, CO 2 welding), various process parameters (welding current, arc voltage, welding speed et al.) have different aspect of influence. The welding current mainly acts on the welding wire deposition speed and the weld penetration depth. When the welding current increases, the welding wire deposition speed and the weld penetration depth increase accordingly. Conversely, when the welding current becomes smaller, the deposition speed of the welding wire and the penetration depth of the weld become smaller. When setting the welding current in the welding process, it is necessary to select the corresponding matching welding current range according to the parameters of the selected welding wire and the size of its diameter. Only within an appropriate current range can the stability and quality of the welding be ensured. In order to verify the effectiveness of prediction of different models, the experimental results of the four methods are compared with the theoretical results. Table 2 shows the prediction results of bead geometry parameters by BP neural network. Table 3, Table 4 and Table 5 show the results by CatBoost, XGBoost and CNN. In order to more intuitively compare the quality of each model in predicting the bead geometry parameters, the bar charts are shown in Figure 5. The vertical axis of each graph represents the percentage of prediction error. The automatic welding is one of applicative implementation, so the prediction range are wider than other problems of prediction, so the experimental errors seem higher than other problems. We just compare the behaviors of four learning methods. For the prediction of three bead geometry parameters, CNN performs best, followed by Cat-Boost and then XGBoost.
According to the experimental results, different models should be selected to make an optimal prediction. However, in general, ensemble learning algorithms perform better in predicting welding process parameters, and CNN performs better in predicting bead geometry parameters.
Another group of experiments was carried out to predict the welding process parameters by using bead geometric parameters, we still use four learning methods, BP neural network, CatBoost, XGBoost and CNN, respectively, and the experimental results are shown in Table6, Table 7, Table 8 and Table 9. For intuitive comparison, Figure 6 displays the effectiveness comparison among four models. Figure 6       shows the comparison between the predicted current (I) and the actual results of the four methods. It can be seen that XGBoost performed best, and then followed by CatBoost. It can be seen from Figure 6(b) that BP has the best prediction effectiveness of parameter of voltage (U), followed by model XGBoost. Figure 6(c) shows the comparison of welding speed (V1) between the actual results of the four methods, and we can find that XGBoost has the best prediction effectiveness, BP performed the second. It can be seen from Figure 6(d) that CNN has the best prediction effectiveness of parameter of wire feeding speed (V2), followed by model BP.

V. CONCLUSION
The intelligent welding model can improve the intelligence and efficiency of robot welding process planning. This is helpful to solve the problem that welding workers are difficult to operate in special welding environment. In this paper, a variety of learning methods are explored to predict the process parameters in the multi-layer and multi pass welding process. With the support of the machine learning and ensemble methods, the welding process parameters and the bead geometry parameters can be predicted mutually. The experimental results show that different learning algorithms have advantage on different kind of tasks of prediction. In the applicative welding process, appropriate model can be selected to realize the intelligence and automation of robot welding.