Neural Network Enhanced Robot Tool Identification and Calibration for Bilateral Teleoperation

In teleoperated surgery, the transmission of force feedback from the remote environment to the surgeon at the local site requires the availability of reliable force information in the system. In general, a force sensor is mounted between the slave end-effector and the tool for measuring the interaction forces generated at the remote sites. Such as the acquired force value includes not only the interaction force but also the tool gravity. This paper presents a neural network (NN) enhanced robot tool identification and calibration for bilateral teleoperation. The goal of this experimental study is to implement and validate two different techniques for tool gravity identification using Curve Fitting (CF) and Artificial Neural Networks (ANNs), separately. After tool identification, calibration of multi-axis force sensor based on Singular Value Decomposition (SVD) approach is introduced for alignment of the forces acquired from the force sensor and acquired from the robot. Finally, a bilateral teleoperation experiment is demonstrated using a serial robot (LWR4+, KUKA, Germany) and a haptic manipulator (SIGMA 7, Force Dimension, Switzerland). Results demonstrated that the calibration of the force sensor after identifying tool gravity component by using ANN shows promising performance than using CF. Additionally, the transparency of the system was demonstrated using the force and position tracking between the master and slave manipulators.


I. INTRODUCTION
T ELEOPERATION indicates the remote control of a slave manipulator by a human operator at a remote site [1]. Its application has been popular in various areas, and a substantial recent advantage is provided by medical applications such as Robot-assisted Minimally Invasive Surgery (RA-MIS). In particular, bilateral teleoperation draws many research interests because it provides haptic feedback for the surgeon, which can ease and improve the surgical tasks performing, for example, enhancing surgical accuracy [2], optimizing dexterity and minimizing the trauma of the patient [3] [4], etc..
In recent years, researches have proved that it is helpful for the surgeon to perceive the sensation of touch from the remote surgical site [5] [6] [7]. The lack of haptic feedback in teleoperated surgical tasks could lead to some adverse effects such as tissue damage, and the operational procedure can be time-consuming. [8]. Hence, achieving accurate force sensing on the remote site is of vital importance for bilateral teleoperation in the robot-assisted surgery using adaptive compensation [9] [10] [11].
The demand for measuring interaction forces in RA-MIS [12] has drawn a lot of interest due to the requirements of the clinical application with bilateral teleoperation [13] [14]. To sense the interaction force on the remote sites [15], force sensors need to be mounted between the end-effector of the slave robot manipulator and the surgical tool. In this case, compared to the solution of installation of a force sensor on the surgical tip, it is possible to ignore the influences from the sensor size and to reduce the equipment cost [16]. However, it will introduce unexpected disturbances, and the output of the force sensor acquires not only the interaction force but also the gravity force applied to the tool. In this sense, it is necessary to identify the weight force of the tool and compensate it from the measured value of the force sensor. Moreover, force sensor calibration remains an issue for bilateral teleoperation because the directions of the forces acquired from sensor and robot must be aligned in the same coordination frame [17].
Several approaches have been introduced for gravity compensation using robot joints angles, which depend on the robot kinematic and dynamic model [18] [19] [20] [21] [22]. This problem has been addressed firstly in [19] using the numerical values of gravity and inertia terms estimated by the control software interface FRI. A proportional-derivative (PD) control with online gravity compensation using joint angles is proposed for regulation tasks of robot manipulators in [20]. Nonlinear optimization methods are introduced to extracting feasible robot parameters from dynamic coefficients in [23]. However, the above methods depend on the robot kinematic model and are difficult to transfer to another general industrial robot. Interestingly, the gravity force applied to the robot tool variables with the orientation of the robot. Hence, we intended to model the tool gravity based on the orientation angles. In this way, the gravity force on the tool can be eliminated based on the build model.
In a practical application, force sensor calibration methodologies should be accurate and with a low computational burden [24]. As one of the traditional regression algorithms, least-square optimization method has been widely employed for modeling the mapping relation between the tool gravity applied to the robot tool and the orientation angles of the tool [25]. However, this method is hard to achieve accurate results and is not practical because it requires the full mathematical expression of the model and it ignores the nonlinear characteristics existing in the robot system [26] [27] [28]. In [29], a calibration method was developed by using a pre-calibrated force plate. Although this method makes the calibration method easier, the dismantling of the sensor remains an issue [30]. A fundamental problem with most of the calibration methods [25] in the literature is that they ignore the gravity influence of the surgical tool and the nonlinear disturbances due to the setup of the tool, which affects the force sensing accuracy in the teleoperation system [10] [31] [32].
In this paper, we utilized two different techniques for tool gravity identification based on Curve Fitting (CF) and Artificial Neural Networks (ANNs) to modeling gravity force variance due to tool's weight according to the tool direction in the Cartesian space. CF and ANNs are capable of modeling mapping relationships between multiple inputs and outputs [33]. The performance of CF and ANNs in terms of modeling accuracy is implemented and compared with a real robot application using a serial robot (LWR4+, KUKA, Germany). Afterward, the force sensor calibration is achieved with the classical singular value decomposition (SVD) algorithm. Finally, a bilateral teleoperation demonstration is performed to show the transparency of the developed teleoperation system by integration with a haptic manipulator (SIGMA 7, Force Dimension, Switzerland). The main contributions of this paper comprise: 1) A novel method is proposed to model the tool gravity using the orientation of the robot. 2) A mathematical model is implemented using CF, and a comparison with an ANN-based method is utilized to show the feasibility of the proposed method with real experiments. 3) A bilateral teleoperation scenario is introduced to demonstrate the efficiency of the proposed identification and calibration method. The rest of the paper is organized as follows: the kinematic model of the serial robot is shown in section II, and the corresponding methodologies are presented in section III, separately. Section IV presents the experiment validation and results of the proposed methodology evaluated with the KUKA LWR4+ robot. Finally, section V concludes and delineates avenues for further work.

II. KINEMATIC MODEL OF THE SERIAL ROBOT
In this paper, the kinematic model of the 7 Degrees-of-Freedom (DoFs) Light Weight robotic arm (LWR4+, KUKA, Germany) is defined using the Denavit-Hartenberg (D-H) parameters [34] as presented in [33] [35]. Figure 1 shows the robot in its home position with a force sensor mounted between the end-effector and the surgical tool. The corresponding D-H parameters can be found in our previous work [33]. According to these parameters, a homogeneous matrix i i−1 T determines the transformation between two consecutive link frames of the serial robot arm from i − 1 to i .
The transformation matrix from the robot base frame to its FIGURE 2: Transformation involved in teleoperated surgical systems. Transformation matrix ( o m T ) from the master reference frame to the slave tool reference frame is adopted to couple the motion between the surgical tool and the master manipulator. e m T is a constant transformation matrix between m and o , which depends on the actual setup of the platform. e o T is the transformation matrix obtained from forward kinematics. w c T is the transformation matrix between c and w , representing the transformation between the endoscopic camera and the operational target.
end-effector frame can be obtained with the forward kinematics. Moreover, the tool pose can be obtained by multiplying the link transformation matrix, as follows [36]: where i i+1 T is the transformation matrix, as shown in (1).

III. METHODOLOGY
In the practical teleoperation applications, the master haptic manipulator and the slave robot are different in kinematics characteristics and workspace size [37], which requires workspace mapping technique to allow the surgeon to span the whole workspace of the slave manipulator [38]. Moreover, its gravity terms due to the weight of the tool should be estimated and compensated during the motion to reproduce only tool-environment interaction force on the hand of the surgeon. Two techniques have been performed to achieve tool gravity identification: CF and ANNs, separately. Also, the performance of both are compared on the same operational procedure in terms of accuracy. Once the tool gravity force is identified, force sensor calibration using SVD is implemented to transform the force information into robot coordination frame to achieve accurate force feedback measurement from the remote site to the local site. Based on the above methodology, a final bilateral teleoperation demonstration for surgical tasks is used to verify the proposed methodology.

A. WORKSPACE MAPPING
Workspace mapping, shown in Figure 2, is performed to map the motion trajectories of the haptic manipulator into a reachable workspace for the slave robot on the remote site. As shown in Figure 2, the slave robot is controlled by the master manipulator through teleoperation. An interpolation method is introduced to enable the robot to achieve the desired position X d ∈ R 3 from the end-effector position X ∈ R 3 following the master manipulator smoothly [39]: where e m T = o m T e o T is the transformation matrix from the master frame to the slave tool frame, as displayed in Figure 2. X r ∈ R 3 is the motion trajectory of the master manipulator. k 0 > 0 is a positive constant coefficient.

B. TOOL GRAVITY IDENTIFICATION
The force sensor is mounted between the slave end-effector to transmit the interaction force from the slave robot to the surgeon, as exhibited in Figure 3. The tool is attached to the force sensor, which measures the interaction force with the environment.  As discussed before, the force sensor cannot measure the interaction forces due to the influence of the gravity force of the robot tool, which varies with the tool direction. It means that the surgeon cannot accurately perceive the delicate interactions between the surgical tool and organ tissue in the patients' abdomen. Therefore, tool gravity identification for the force sensor is required to compensate for the influence of the tool weight. The force acquired by the force sensor can be expressed as follows: where F T oolGravity ∈ R 3 is the force generated by tool gravity, while F Interaction ∈ R 3 is the interaction forces between the tool-tip and the organ tissue in the abdomen. It should be noticed that the tool's gravity force on the force sensor is varying with the robot arm placement, in particular with the orientation of the tool. During the whole procedure, the orientation of the tool is the only variables that can influence the output of the force sensor. It is essential to consider gravity identification depending on the orientation of the tool. The relation can be modeled based on the orientation of the tool and its corresponding force sensor output.

1) Curve Fitting based Tool Gravity Identification
One of the most efficient techniques to model the mathematical equation and to predict the unknown values is the CF. In the related works, the Least-Squares method has been introduced to fit the data mapping frequently. [40].
The mathematical model of the influence of the gravity force on the force sensor, shown in Figure 4, depending on the tool orientation, can be defined as follows: where F x , F y and F z are the outputs of the force sensor. The unknown constant parameters are the mass m, the coefficients a, b, and c. g represents gravity which is 9.8 m/s 2 . d is a deviation error angle around z axis from the tool installation. The angles θ 1 and θ 2 are computed from the orientation angles of the tool pose, shown in Figure 4. It should be noticed that the relation between the angles θ 1 , θ 2 , and the Euler angles θ x , θ y , θ z is a mapping relation, where θ 1 is determined by θ z while θ 2 is determined by θ x and θ y . If an enough data set is collected, the CF is capable of estimating accurately the constant parameters existing in Eq. (4). However, due to the installation of the tool in the practical application cannot be in a straight way, there should be a deviation error on θ 1 . Hence the proposed mathematical model is not able to project the gravity force of the tool on the force sensor in an accurate way. As a parametric regression, the accuracy of CF relies on the prior knowledge of the mathematical system model, while it is difficult to obtain an accurate model considering the mechanical error. Hence, it is difficult to achieve precise mapping between the gravity force and the orientation angles.

2) ANN-based Tool Gravity Identification
Neural Networks (NNs) [41] provide a new solution for the modeling of linear and nonlinear curve fitting problems [42], which does not require the dynamic model of the system and has the learning capacity to model any complex function and nonlinear relationships. [43].
The force value of the force sensor varies due to the orientation of the tool. Since the regression model between the tool orientation and the force projected on the force sensor has been introduced and solved using CF, the regression function can be redefined by mapping the Euler angles (θ x , θ y , θ z ) to the projected force on the force sensor (F x , F y , F z ) can be defined as: It is known that ANNs is capable of approximating any function, regardless of its linearity. In this paper, a feedforward back-propagation ANNs with one hidden layer was implemented to train the regression mapping function. According to the input and prediction output, Figure 5 portrays the diagram of the proposed NN to estimate the mapping relation, where the inputs θ x , θ y and θ z represent Euler angles of end-effector and F x , F y and F z are the target outputs of our ANNs model.
The performances of the regression network was determined by the number of neurons of the hidden layer In this paper, the nonlinear least-squares algorithm Levenberg-Marquardt algorithm is adopted to calculate the maximum or minimum gradient. It has the local convergence of the Gauss-Newton method to minimize those functions and has a gradient descent method of global characteristics to look for a new search direction. The training of the neural network and the performance index is set as the mean square error. The final NN model can be written as: is the bias matrix of the first layer, j is the neuron numbers and B 2 ∈ R 3×1 is the bias of the output layer, W 1 ∈ R j×3 and W 2 ∈ R 3×j are the corresponding weight matrix. The initial condition of the weights and bias are initialized to a small random number. In this paper, parameters move in the opposite direction of the error to reduce the mean square error to get the minimum value. The updating law to determine the weight matrix adopted the increment way. The updated law of the weight is given by Gradient Descent rule using the following equation: where, η is the learning rate and L is the loss function. In this article, η = 0.01. To improve the effectiveness for training the ANNs model, we choose 1.05 ratio to increase learning rate and set the maximum validation failures is 6. The training set was used to update the weights of the neurons with the predefined number of iterations. When the NN converged to its final configuration, the testing set was used to assess its actual ability to predict force sensor outputs based on Euler angles.

C. FORCE SENSOR CALIBRATION
The force sensor is a particularly significant source of feedback in robotic applications to measure forces along x, y and z axes at robot's end-effector in order to increase the sensitivity of the surgeon [44]. Force sensor should be calibrated in the system to project the measured force into the robot reference frame. SVD has been successfully applied to a wide variety of domains [45] to solve linear algebra transformation. In this paper, SVD [46] is adopted to figure out the transformation (calibration) matrix f e T between the slave's end-effector and force sensor, reference frames, as depicted in Figure 6. Figure 7 demonstrates the input-output of SVD calibration method, where F R ∈ R 3 , F S ∈ R 3 , are the robot and sensor forces, respectively. f e T ∈ R 4×4 is the obtained calibration matrix.

D. BILATERAL TELEOPERATION
The control framework that can offer such a capacity in robot-assisted surgery is called bilateral control which means the exchange of information (position, velocity, and force) between master and slave manipulators bi-directionally in real-time via a communication network [47], as demonstrated in Figure 8. Bilateral teleoperation can bring the robotassisted surgery to a new level with furnishing significant precision and improvement with dexterity even in a minimally invasive manner.

A. SYSTEM OVERVIEW
A brief description of bilateral teleoperation system developed in this project is shown in Figure 9. A redundant robot (LWR4+, KUKA, Germany) serves as the slave manipulator torque-controlled through Fast Research Interface (FRI), which could provide direct low-level real-time access to the robot controller (KRC) at rates of up to 1 kHz [15]. The teleoperation scheme implements bilateral teleoperation control with a master device (Sigma 7, Force Dimension, Switzerland) and a switch pedal [17]. The software system was developed with OROCOS (Open Robotic Control Software, http://www.orocos.org/) application with a real-time Xenomai-patched Linux kernel and ROS (Robot Operating System, http://www.ros.org/) kinetic in Ubuntu. ROS vision node and OROCOS torque controller were executed on separate computers with UDP communication between each other to guarantee control frequency: the control loop and VOLUME 4, 2016 FIGURE 8: Typical Teleoperated Surgery Framework. The surgeon sitting on the local site uses the master manipulator and foot pedal to drive the motion of the tool of the surgical robot (slave) from the actual point X to the desired point X d . F m is the force applied to the master manipulator. X r ,Ẋ r ∈ R m are the desired Cartesian position and velocity in the master frame, respectively. F s is the force applied to the slave manipulator. X ∈ R m ,Ẋ ∈ R m are the actual Cartesian position and velocity in the slave frame, respectively. F e is the external force on the end-effector from the environment. the vision ROS node was executed on separate computers. The system consists of: • a 7 DoFs master haptic interface (Sigma.7, Force Dimension, Switzerland) and a foot pedal [48], implementing a 3D Cartesian position control. • a 7 DoFs LightWeight robotic arm (LWR4+, KUKA, Germany) as a slave device. • a 6-axis force sensor (M8128C6, SRI, China) [49] that has the purpose of measuring interaction force between the surgical tool-tip and the environment. • a HD endoscopic camera which allows the surgeon to view the environment. To conduct the identification procedure, two experiments were performed to acquire the data: the training dataset (41729 samples) and the testing dataset (32047 samples). Then we applied the obtained identification models online to eliminate the gravity force of the tool separately. For each method, we collected data for calibration (4164 samples for CF and 4115 samples for ANNs). The detailed procedure and results are shown in the following parts.

B. CALIBRATION AFTER CURVE FITTING IMPLEMENTATION
The first method used for tool gravity identification is CF. As mentioned in section III, gravity identification is implemented concerning current end-effector orientation. Firstly, hands-on control is activated to allow the user to move the robot arm without touching the robot tool and the force sensor. In this way, two groups of data are collected for estimation and validation, as depicted in Figure 10. By utilizing the CF technique, the constant unknown parameter m, and the constant coefficients a, b and c can be obtained the first group of sampled data (41729 samples). Then, the obtained parameters are placed in the mathematical model to predict the force on the force sensor, which is expressed as follows: F x,estimated = −0.3434 * g * sin (θ 1 ) * cos (θ 2 + 1.401) +0.6 F y,estimated = −0.3434 * g * sin (θ 1 ) * sin (θ 2 + 1.401) +1.1 F z,estimated = 0.3434 * g * cos (θ 1 ) + 2.0 (8) In order to verify if the obtained model (8) is able to fit the real measurements acquired by the force sensor, the second groups were used for validation (32047 samples) The error between the real force and estimate force is analyzed. Figures 11 and 12 depict the difference between the real and estimated force along the different axis. After the gravity force F T oolGravity is identified using CF technique, force signals data are acquired from both robot and sensor, as shown in Figure 13, to perform force sensor calibration so as to couple both signals by means of SVD method. Results of calibration are exhibited in Figure 14.

C. CALIBRATION AFTER ANNS IMPLEMENTATION
Except for using CF, ANNs is also utilized to modeling the tool gravity force. As mentioned in Section III, the number of neurons in the hidden layer is determined by assessing the performances of the regression networks. We adopt a different number of neurons to train several ANNs models. The root mean square error (RMSE) is used to evaluate the   performance of the ANNs model, which can be calculated by Equation 9.
whereF is the predicted force and F is the actual measured force (real value). i is the order of input and output sequences, and n is the total sampling number.   Table 1 enumerates the results of the average obtained three RMSEs on the training and testing datasets, namelȳ In the training procedure, the average of RMSE is reduced along with the increase in the number of neurons in the VOLUME 4, 2016   hidden layer. However, in the testing processing, the changes of mean RMSE reaches to the lowest value (i.e., 0.1030 N) when the number of neurons is 30, then becomes worse. This phenomenon is caused by the under-fitting and over-fitting of the ANNs algorithm. So, the best ANNs regression model is the one with 30 neurons in the hidden layer. Figures 15 and 16 show predicted force curves by the ANNs model (30 neurons) and real forces on the training and testing dataset, respectively.
As mentioned above, after identifying tool gravity component by ANNs, we perform the force sensor calibration again by collecting another data (see Figure 13) and then applying the same method SVD. Results of calibration are depicted in Figure 17.

D. DISCUSSION
As mentioned above, CF and ANNs regression methods were used for tool gravity identification. Furthermore, force sensor calibration based on SVD method was applied after tool gravity compensation. The comparison verification has been implemented to validate the best efficient model for tool gravity identification. Table 2 lists the tri-axis RMSEs computed by CF and ANNs (30 neurons) methods after tool gravity identification. The 'overall' row is the sum of obtained three RMSEs.  The calibration errors prove that the regression performance on the testing set is worse than the training set. Notably, the CF model almost loses the fitting with an overall error of 5.684N . The ANNs model can also map the Euler angles to the forces with a lower overall RMSE 0.309N , which is close to the above results validated on the training set. Although the obtained errors prove that the ANNs model (30 neurons) is the best method to predict new forces, the CF model will be a fast method for predicting the forces. Because it has a simple regression function and fewer parameters to calculate. So, if the accuracy is not a compulsory requirement, the CF model can save some computational time for regression.
Attaining the two issues of gravity identification and sensor calibration, this enables us to achieve transparency in teleoperated surgery. Figure 18 shows the force tracking between master and slave devices, while Figure 19 shows Cartesian positions tracking. It demonstrates that the proposed methodology could achieve transparency in teleoperated surgery.

V. CONCLUSION AND FUTURE WORK
In this paper, the ANNs model enhanced method is presented for surgical tool gravity identification and force sensor calibration in bilateral teleoperation with force sensing. Firstly, the tool gravity force was identified by CF and ANNs, separately. Afterward, force sensor calibration was implemented using SVD. Results from the comparison show that residual error of the calibration of force sensor after tool gravity identification is improved more than without tool gravity identification. By comparison, ANNs is able to model the mapping relations without the prior knowledge of the mathematical model between the orientation angles and the force. However, since it is a non-parametric 'black box', it lacks the knowledge of the dynamics of the system, and its accuracy performance may be worse than the CF which is based on the mathematical model. Furthermore, the comparison errors prove that ANNs methods can get higher accuracy than the CF model. So the ANNs model with 30 neurons in the hidden layer is the best model for predicting new forces. A preliminary practical experiment has been performed to investigate the transparency performance of bilateral teleoperation. It can be concluded that the proposed tool gravity identification and calibration method for bilateral teleoperation can be easily replicated on other general robots since the model depends on the rotation of the robots instead of the robot joints angles. Future work will consider more challenging problems (e.g., dead-zone and time-delay) in our the bilateral teleoperation control framework. The system stability and tracking accuracy might not be guaranteed under these situations, which are a precondition for safety in a surgical operation. ALESSANDRO RIZZO (M'01-SM'08) received the Laurea degree (summa cum laude) in computer engineering and the Ph.D. degree in automation and electronics engineering from the University of Catania, Italy, in 1996, and 2000, respectively. He is an Associate Professor at Politecnico di Torino, Italy, where he is engaged in conducting and supervising research on cooperative robotics, complex networks and systems, modeling and control of nonlinear systems. Since 2012, he has also been a Visiting Professor at the New York University Tandon School of Engineering, Brooklyn, NY, USA.
He is the author of two books, two international patents, and more than 130 papers on international journals and conference proceedings. Prof. Rizzo has been the recipient of the award for the best application paper at the IFAC world triennial conference in 2002 and of the award for the most read papers in Mathematics and Computers in Simulation (Elsevier) in 2009. Prof. Rizzo is also a Distinguished Lecture of the IEEE Nuclear and Plasma Science Society. More details can be found at the website staff.polito.it/alessandro.rizzo.