Development of a Neural Network-Based Control System for the DLR-HIT II Robot Hand Using Leap Motion

In this paper, a neural network (NN) based adaptive controller has been successfully developed for the teleoperation of a DLR-HIT II robot hand using Leap Motion sensor. To achieve this, the coordinate positions of the human hand fingers are captured by a Leap Motion sensor. Moreover, inverse kinematics is used to transform the Cartesian position data of the fingertips into corresponding joint angles of all the five fingers, which are then directly sent to teleoperate the DLR-HIT II hand via User Datagram Protocol (UDP). In addition, a NN-based control system programmed by the MATLAB Simulink function has been investigated to enhance the overall control performance by compensating the dynamic uncertainties existing in the teleoperation system. The stability of the control system has been proved mathematically using Lyapunov stability equations. A series of experiments have been conducted to test the performance of the proposed control technique, which has been proved to be an effective teleoperation strategy for the DLR-HIT II robot hand.


I. INTRODUCTION
In recent years, there has been a rapid advancement in robotics, with its applications growing extensively in numerous fields. Development in robotics has greatly facilitated people's lives and is one of the most rapidly emerging industries of today's daily life and manufacturing usage. Most countries are paying much attention to the development of robots, such as the National Robot Program of the United States, the Horizon 2020 of the European Union and the National Natural Science Foundation of China's 863 Programme [1]. From the beginning of the Industrial 4.0 revolution, the role of robots has become increasingly significant with the advance of science and technology. However, this role remains largely limited to traditional industrial applications such as pre-set and repeated tasks. At present, it remains a challenge to develop robots that are capable of performing an assortment of tasks in unknown, complex and dynamic environments [2]. Besides, service-oriented, medical as well The associate editor coordinating the review of this manuscript and approving it for publication was Ming Luo . as industrial robots are required to work in complex external environments. Hence, control of robots in unknown and dynamic environments has been widely studied and a higher demand for advanced intelligence in robots has been recognized. It is noteworthy that in the working environment, robots are inevitably affected by unpredictable and uncertain external disturbances. In this regard, traditional control methods are not sufficient for qualifying them in terms of intellectuality. Recent research has been actively investigated into the interaction and control between robots and external forces. Interactive control technology cannot only assure compliance of robot behaviour, but also provide a relatively safe operating environment for operators, which are impossible to achieve through traditional methods of robot control.
According to [3], the human-robot interaction (HRI) is a sub-area of the human-computer interaction (HCI), which studies the interaction between humans and robots, and focuses on developing more intelligent and anthropomorphic robots. HRI is widely used in research and implementation of robot systems in hazardous zones where human involvement needs to be minimized and remote operation of robots is required. These inventions can also apply to care for the elderly and the disabled personnel, as well as to entertainment purposes [4]. There are various methods of human-robot interaction, amongst which, two are considered the main methods of -physical interaction and teleoperation interaction. For specialized tasks, such as medical operations, the safest method for human-robot interaction is teleoperation [5]. In this method, the user interfaces with the robot through a haptic signal for coordinated feedback. The research [6] carried out experiments to characterize this problem, and proposed several methods to provide haptic feedback in order to improve a surgeon's performance. However, they do not take the enough required control room for the surgeon into consideration. As already known, the surgeon's activities have high requirements for cleaning, and thus the remote control is a better choice. In the research [7], a research team introduces a visual impedance scheme for vision-based control of the robot so as to realize task-level dynamical control. In the control scheme, the image features are applied to the impedance equation so that integration of a visual servo and a conventional servo system could be accomplished.
The interaction between robots and complex environments can lead to potential instability, inaccuracy, mechanical nonlinearity and saturation, thus may have a negative impact on the overall working system. At present, there are two main methods of interactive control between robot and environment external forces namely, the hybrid force/position control and impedance/admittance control [8]. Hybrid force/position control has the potential of achieving good control performance and anti-disturbance ability. However, in case of a strong interaction between the robot and the environment, the hybrid force/position control method often leads to system instability and security problems [9]. As compared to hybrid force/position control, impedance/admittance control aims to establish the correct balance in the relationship between the robot and interactive force received from the external environment. Through impedance/admittance control method, robot's behaviour can be adapted to the environment and the overall stability of the control system can be ensured [10]. Therefore, further study and development of robot interactive control technology for dealing with various uncertainties in a real-world situation is necessary so as to ensure that robots can work in a dynamic and unknown environment. Hence, the development of robot interactive control technology is of great practical significance.
There are dynamic uncertainties in a robot's control system or during the modelling which have consequences in the learning and reproducing performance. These uncertainties can be controlled with the development of controls using Fuzzy Logic or Radial Basis Function (RBF) Networks. RBF is a three-layer forward network with a single hidden layer [11]. Its first layer is an input layer and consists of signal source nodes, and its second layer is a hidden calculation layer. The total number of hidden layer nodes depends on the complexity and requirements of the problem described. The transformation function of the neurons in the hidden layer, for example the radial basis function, is a non-negative linear function that is radially symmetric and attenuated to the center point [11]. RBF is a local-response function, and the specific local-response is reflected in the transformation of its visible layer to the hidden layer, a phenomenon which differs it from other networks. The third layer is the output layer which responds to the input mode, and therein the input layer only acts as a transmission signal layer [12]. Previous forward network transformation functions were used for a global response. Functionally, the input layer and the hidden layer can be connected with a weight factor. The output layer and the hidden layer perform different tasks, hence learning strategies used for them will also be different [13]. The output layer adjusts to the linear weight using a linear optimization strategy, so the learning speed becomes faster. However, the hidden layer adjusts parameters of the activation function (Green's function and Gaussian function, more commonly the latter) through nonlinear optimization strategies which make the learning process slower [14].
Recently, researches on the dexterous robot hands have been widely investigated. For shortening the distance of power transmission, the embedded actuator method has been used to integrate all the actuators inside the hand, fingers or joints. The Gifu Hand [15], the DLR Hand II [16] and the DLR-HIT Hand I [17] have taken this method for their robot hands products. According to [18], the Stanford-JPL Hand, the Utah/MIT Hand, the NASA Robonaut Hand and the DLR Hand Arm System are designed with a different method, where all the actuators are mounted outside the robot fingers and driven through tendon cables. In this paper, the modelling VOLUME 7, 2019 and control schemes of the dexterous robot hand DLR-HIT II (shown in Fig.1) have been demonstrated, which is a multisensory five-finger robot hand with a total of 15 DOFs and more than 100 sensors produced by German Aerospace Center (DLR) and Harbin Institute of Technology (HIT).
Commonly, the robot hand alone is not widely studied and the research works only on a robot hand itself are limited comparing to those on the robot manipulators. Some research institutes and industries apply the robot hands to a robot manipulator for their studies and operations to do some specific tasks, such as the pick-and-place tasks with grasping and opening actions of the robot hands [19]. In this case, the robot hand is implemented as an overall object so that its control method normally only contributes to the clenching & unclenching of the robot hand. In addition, when a robot hand is used to grasp a heavy object, the gravity and inertia can cause problems to the control system. In this paper, a novelty robot hand tele-control system mixed with an adaptive function approximate method has been developed, where all the fingers can be separately controlled with overcoming of the unknown dynamics.

II. HAND GESTURE RECOGNITION USING LEAP MOTION
In this paper, the Leap Motion somatosensor (show in Fig.2) produced by Leap Motion Inc., a US-based somatosensory controller manufacturer, released on February of 2013 was employed for tele-control purposes. It features a natural gesture function to control computer programs that replaces the need for traditional mouse and keyboard controls [20]. By acquiring user gestures in spatial motion data, we can naturally manipulate 3D objects using familiar gestures. The Leap Motion was a major breakthrough in the accuracy, efficiency, and portability aspects, where the accuracy of the Leap Motion is up to 200 times greater than other somatosensory products such as Microsoft's Kinect [21]. It has been applied in many human-computer interaction roles such as education, medicine, games, and Google Maps 3D due to its intuitive nature and depth of control that can allow for circumvention of the requirements for a mouse or touch screen.
Internally, a Leap Motion device consists of two narrow-band filters, two mid-low-pixel cameras, three infrared LEDs, and a USB high-speed chip in [22]. The principle operation of Leap Motion is to capture a hand image illuminated by the infrared LED lamp through two cameras and establish a three-dimensional model of the hand by analysing the change of gesture in the stereo camera images. When the hand enters the field of view of the device, the stereo camera system begins capturing the target and calculates the parallax of the target in real time so that its spatial information is obtained [20]. The built-in communication chip in the body uses standard USB transmission technology to digitise the collected image information, and transmit the converted data to the computer. The computer then restores the gesture changes through image recognition and calculation and resubmits hand motion to a PC application for the usages by the programmer. Fig.2 above shows the system configuration of this research. Here the Leap Motion is used to collect the fingertips' data in Cartesian space and then those data are transmitted to corresponding angle values by applying the inverse kinematics calculations. In addition, a remote master PC connects to the DLR-HIT II robot hand's QNX controller by an Ethernet cable, where the NN-based adaptive controller is running on the Simulink of the MATLAB. The methodologies for our proposed teleoperation system are demonstrated in the following sections.

III. ACQUISITION OF FINGER JOINT ANGLES A. HAND MODELLING
In general, the model of the human hand can be thought of as 5 mechanical rigid body models, which are representing 5 fingers respectively. The index finger, middle finger, ring finger and the ring finger all consist of a proximal interphalangeal (PIP) joint, a distal interphalangeal joint (DIP), and a metacarpophalangeal (MCP) joint. The thumb differs, only consists interphalangeal (IP) joint and MCP joint. In addition, there are five carpometacarpal (CMC) joints in the palm. According to the characteristics of kinematics, the thumb is composed of three degrees of freedom (DoF), and the three joints are all rotating joints, while all the other fingers have 4 DoFs. This paper builds a model of the finger by establishing a Denavit-Hartenberg (DH) model of the human hand, which is a method proposed by Denavit and Hartenberg in 1955 to describe the relationship between the joints and the connecting rods of the robot arm using the geometry of the system [23]. This method fixes a coordinate system on each link of the modelled robot. The homogeneous transformation is used to obtain the spatial relationship between the adjacent two links. The sequential transformation can finally derive the pose of the end effector relative to the base coordinate system to establish the kinematic equation of the robot. This paper will present the method of building a robot model and construct a dynamic equation. Fig.3 shows the DH model representation and its associated parameter definitions.
θ i indicates the angle between x i−1 and x i along the z i−1 direction; d i indicates the distance between x i−1 and x i along the z i−1 direction; α i indicates the angle between z i−1 and z i along the x i direction; a i indicates the distance between z i−1 and z i along the x i direction. In general, d i , a i , and α i are constants, and for a rotating joint, θ i is a joint variable. According to the DH modeling method, combined with the structure of the human finger, the kinematic model of the human hand is given as shown in Fig.4.
There is no any motor in the robot hand palm so that we only need to take the robot hand fingers into account. In Fig.4, we use the ''o'' point as the center point of the first knuckle and establish a coordinate system. In the ''o'' coordinate system, the direction of the axis z is perpendicular to the palm plane upward, and the direction of the axis x is upward. According to the structural difference, here we define that there are two cases to be modelled, which are the thumb and the rest. L i represents the distance between each finger joint. Taking the ring finger as an example of ''rest'', the corresponding DH parameters for the two above mentioned cases are given in Tab.1 and Tab.2. Once the DH parameters tables are obtained, the model of the whole hand can be successfully established.

B. VECTOR APPROACH BASED INVERSE KINEMATICS
The vector approach based inverse kinematics is used to calculate the joint angles from the end effector's coordinate location, which can be obtained directly from the Leap Motion SDK. It returns the position of every single fingertip in Cartesian space, which can be converted to the required joint angles of fingers (θ 1 -θ 4 ), which will be used for the neural networks aspects later. The distance between two points in the 3D space can be expressed as: where (x 1 , y 1 , z 1 ) and (x 2 , y 2 , z 2 ) are points in the 3D space, d 3D is the length between these two points. Using the angle formula of the space vector: where the a· b can be obtained by giving the spatial coordinate information of Cartesian coordinates of four degrees of freedom of the three joints of the human finger returned by the Leap Motion SDKs; | a|· b can be calculated by equation (1). Thus, all the k fingers' joints angles can be obtained using the vector approach based inverse kinematics methodology.

IV. NEURAL NETWORKS BASED CONTROL SYSTEM
In adaptive control, a neural network is a commonly used function approximation tool. The unknown function term in the control system is estimated by the neural network. A neural network can be grouped into linear parameterization and nonlinear parameterization, corresponding to RBFNNs and multi-layer neural network (MNNs). RBFNN not only is simple in structure and fast in learning, but also avoids the problems of many nerve layers. As a result, the RBFNN meets the real-time requirements of the control system. Therefore, in this paper, the RBFNN is used to approximate the unknown terms of the control system. In control engineering, the universal approximation property of RBF neural network has been widely used as a function approximator. RBF neural network consists of an input layer, a hidden layer and an output layer. The input layer to the hidden layer is a nonlinear transformation, and the output layer is a linear combination of the output of the hidden layer. Therefore, the RBF neural network belongs to the linear parameterized network. In this paper, a continuous function can be approximated using linear parameter RBF neural networks, such as F(z) : R m → R, over a minimized set z ⊂ R m . This can be formulated as, in [25]: where W * = [w * 1 , w * 2 , · · · , w * l ] T ∈ R l is the weight vector, z ∈ z is the input vector with z ⊂ R m being a minimal set, l is the NN node number, and z is the estimation error. S(z) = [S 1 ( z − µ 1 ), · · · , S l ( z − µ l )] T , is the regressor vector, with a radial basis function S i (·), and µ i (i = 1, · · · , l) a central inside S i (·). The Gaussian functions are presented as follows: where ς is the variance and µ i = [µ i1 , µ i2 , · · · , µ im ] T ∈ R m represents the center of each receptive area. The notation of the data output from the result of the inverse kinematics, in addition to the desired joint space trajectory of the i th finger q di is defined as: Here, applying the NN control procedure to achieve the accurate tracking of the reference joint trajectory is used. Hence, the controller should be fast respond, stable and accurate. The reference joint trajectory of the i th finger of the robot hand q di ∈ R 4 consists of a time series of the angle values that are generated from the motion data obtained by the inverse kinematics. The angular matrix q i ∈ R 4 denotes the actual joint positions of the i th finger during the teleoperation. At that point, as indicated by [25], the dynamics of the robot hand's i th finger are given as (6).
where M (q) ∈ R 4×4 is the inertia matrix, C(q,q) ∈ R 4×4 is the Coriolis matrix, G(q) ∈ R 4×1 are the gravity terms and U (q) ∈ R 4×1 are unmodeled elements owing to the system uncertainties.
Then the closed circling framework dynamic equation is given below as (10) [25].
The RBFNN based approximation approach is applied as follows [25].
where W * M , W * C and W * H are the constant ideal weight matrix; S M (q), S C (q,q) and S H (q) are the basis function matrix, and M , C and H are the mismatch uncertainties due to the fact that the number of the hidden neuron is limited.
Then the equation for basis function matrix can be written as follow [25], where The estimates of M (q), C(q,q) and H (q) can be obtained as (14).M Now, substituting (14) into (10), then the previous equations are simplified as  [27].

V. STABILITY GUARANTEE OF CONTROL SYSTEM
Here, the Lyapunov candidate function is chosen as, where Q M , Q C and Q H are positive fixed weight matrix. Then, the derivative of V , which showṡ (17) Utilizing the property:Ṁ (q)-2C(q,q) are the skew symmetric matrix, and equation (17) becomes: The ideal weight matrix W M , W C and W H are constant matrices, the following relationship can be implemented: substituting (15) into the equation (18), then we have: According to [25], the upgraded principle is given as follow,Ẇ where σ M , σ C , σ H are pre-designed positive constants. Combining (21) and (20), then: Applying Young's inequality, (22) can be extended tȯ As indicated by Lyapunov stability theorem, the closedloop stability is guaranteed.

VI. EXPERIMENTAL STUDIES A. EXPERIMENT SETUP
In these experimental studies, series of tests were set up to examine the accuracy of the designed control system as shown in Fig.5, using NN learning. The Leap Motion API & SDKs based detector tracked and provided the coordinate data of all the five fingertips, the inverse kinematics calculated all the corresponding three joint angle values of each finger. UDP was employed to send all the 15 set of finger joint values q d to the NN based adaptive controller written in the Simulink function of MATLAB to control the robot hand. In the first experiment, the operator smoothly moved his fingers by putting his hand over the LeapMotion (shown in Fig.2). At the same time, the angular values of the finger joints were outputted to a local file for recording purposes. The motion space of the DLR-HIT II robot hand is different from the one of LeapMotion. Therefore, we need to do a linear space transformation to project the spatial coordinates of LeapMotion into the spatial coordinates of the robot hand. The linear relationship can be expressed as: where T is the transformation matrix and B is the deviation matrix. The linearly transformed reference trajectory q d served as the reference trajectory of the robot hand and the input signal of the controller. In this experiment, the NN based adaptive controller was not applied. For the second experiment, a heavy object was organized to be put on the area that the DLR Hand II can reach and grasp using 2, 3, 4, 5 fingers respectively. Meanwhile, the operator, stood in front of the robot hand and remotely controlled the robot hand for fingers closing/opening tasks by putting his hand over the Leap Motion sensor and performing grasping/opening behaviours of his hand with 2, 3, 4, 5 fingers respectively. In addition, in this experimental work, we defined the stiffness of each robot hand finger as 500(N /m) because in this level, the objects been grasped was not easily fallen down or neither damaged. The reference trajectory q d was linearly converted and used as the input signal of the controller. The controller based on the RBFNNs function approximation made the actual trajectory of the DLR-HIT II robot hand keep up with the trajectory of the hand of the human operator. In this experiment, the operator completed a periodic set of grasping actions at a steady speed. We mainly tested the tracking effects of the finger's joints without the ones in the palm: PIP joints, DIP joints, MCP joints, CMC joints and IP joints. First, the LeapMotion camera space was transformed into the motion space of the DLR-HIT II robot hand by the linear transformation relationship of equation (25), where more detailed methodology of the coordination system transformation can be found in our previous work [28]. Then, the remote master computer sent the transformed track signal q d to the slave computer through UDP to control the robot hand. In the torque mode of the DLR-HIT II robot hand, the reference trajectories signals q d were used as the input of the controller, which were also the joints angles of real-time transformations from LeapMotion.
Moreover, the joints angles of the robot hand were used as the feedback signals.
In order to better understand the sensitivity of our proposed approach to the parameters and constants, the third experiment was conducted to investigate the sensitivity of the performance relative to all parameters, which are the control gain K; RBFNN parameters Q and σ ; the initial values of the weight matrix:Ŵ T M ,Ŵ T C ,Ŵ T H ; the number of nodes in the neural network l; the variance of Gaussian function ς, respectively. Firstly, the averaged joint angular difference among all the 4 joints on each finger between the robot hand and the human hand was selected as the performance index. Then the changes of the performance index along the parameters tuning was studied to find the corresponding parameter values when the performance index has the lowest value. The recorded data from the first experiment were directly imported into the controller system via MATLAB. We made each parameter changes with the same ratio for 5 times in the vicinity of the nominal value in turn, and kept the other parameters remain unchanged, to observe the change of the performance index curve. Finally, the particular parameter where the control performance is more sensitive to, can be defined. Furthermore, the values of all corresponding parameters can be obtained in the case of the highest curve fit.

B. EXPERIMENTAL RESULTS
For the first experimental result analysis, Fig. 6 illustrates the experimental data of the joint angular variables for the DLR-HIT II robot hand while performing the smooth  movement using all the 5 fingers, which are the robot hand's real trajectory after the calculation of the inverse kinematics. It is easy to find that when the operator maintains a static posture, the trend of each joint angle remains relatively stable. The main reason for some jitter in the curve is that it is difficult for the operator to ensure that the hand is completely static and the data acquisition of the LeapMotion is not completely stable during the data acquisition process. As the comparison, Fig. 7 illustrates the angular trajectory collected from LeapMotion, which represents the finger's joints angular variables for the hand of the human operator. From above two plotted graphs, it can be seen that even there are tracking errors existing, our proposed LeapMotion sensor guided teleoperaion system without NN learning can track the human operator's hand motion, make the DLR-HIT II robot hand imitate the human operator's hand motion in a fair level.
There are dynamic uncertainties existing in the teleoperation process. To overcome this issue, RBFNNs were employed to approximate the unknown dynamics in this paper. During the grasping tasks, it can be found that even the stiffness parameters have been set correctly, using 2,3,4 fingers to stably grasp the objects were not easy to achieve. After plenty of trainings, we noticed that using all the 5 fingers had the best grasping stabilities. Therefore, all 5 fingers were used to conduct grasping tasks for the second experiment. In addition, because the proximal interphalangeal (PIP) joint among all the joints of each finger was most bent while grasping items, here only the PIP joints of all the 5 fingers were taken for data analysis to the adaptive control process in this experiment, for the easy-to-see experimental data curves and analysis as well, which were representing the ThumbUp, IndexUp, MiddleUp, RingUp and LittleUp joints, respectively. For our second experiment, even though all the data have been collected when performing the grasp tasks using 2,3,4,5 fingers, respectively, it is still needed to plot the data set for just the 5 fingers' motion in order to taking the visual duplicate problems into account. Fig. 8 shows the comparison of all the 5 fingers' PIP joints controlled with NN learning and without NN learning. It can be seen that the tracking performance of the fingers' joints have been improved after adopting the NN learning control. As can be seen from Fig. 8, there is external torque applied to all 5 fingers when the DLR-HIT II robot hand grasps the heavy objects. As a consequence, the angular trajectory of the robot hand differs from the one of human operator when the only the LeapMotion controller is employed. While the proposed RBFNNs based adaptive controller is employed together with the LeapMotion tele-control system, in order to adapt to the external torque, the robot hand generates a corrected reference trajectory q r by correcting the previous reference trajectory q d . In addition, the actual curve of the joint can converge to the reference trajectory curve better. The main reason for the error between the curves is the delay of UDP data transmission and the jitter during the movement of the robot hand. Finally, it can be concluded that the controller has made the robot hand accurately track the corrected reference track q r .     According to our large numbers of parameters tuning based on the experimental study, the control system had the greatest performance when the parameters are set as: control gain K = 12; RBFNN parameters Q = 10 3 , σ = 10 −2 ; the initial value of the weight matrix:Ŵ T M =Ŵ T C =Ŵ T H = 0 ∈ R n×l ; the number of nodes in the neural network l = 3; the variance of Gaussian function ς = 100, wherein the initial value of the weight matrix and the NN nodes numbers are set according to the experiences based knowledge from our previous research works.
From Tab.3, we can observe that the performance index (PI), which is the averaged joint angular difference among all the 4 joints on each finger between the robot hand and the human hand, changes slightly when the parameter K is perturbed from 10 to 14; the performance index changes fairly when the RBFNN parameter Q is perturbed from 830 to 1170. From Tab.4, we can observe that the performance index changes fairly when the RBFNN parameter σ is perturbed from 0.008 to 0.012; the performance index changes slightly when the parameter Gaussian variance ς is perturbed from 80 to 120. Overall, the control performance is more sensitive to the RBFNN parameters Q and σ . How to improve the robustness of the proposed control system against these parameters, and reduce the overall sensitivity of the method relative to parameter perturbations will be systematically investigated in our future work.   9 shows the joint compensation torque curves. The output torque of the NN is mainly to compensate for the uncertainties in the system to achieve the desired control performance. It shows that the uncertainty of the control system appears at certain moments after the application of the neural network, and the control system can be well compensated by the input torque. The NN learning weighs can be seen in Fig.10. As the stiffness has been set to a constant value, the angular value difference of finger joints between the operator and the robot hand are selected to assess the control performance. It can be concluded from the above results that NN learning promotes the robustness of the tele-control system as it has an acceptable error margin during the teaching and learning process, thereby satisfying the proposed design goal.

VII. CONCLUSION
In this work, an NN-based adaptive teleoperation system for the DLR-HIT II robot hand has been designed using the Leap Motion hand tracking device. Firstly, the Leap Motion device outputs the 3D Cartesian coordinates of each joint of the operator's hand, then inverse kinematics are used to obtain the relevant joint angles for the robot hand. Next, these joint angles are sent via the UDP communication protocol to the controller which subsequently outputs joint angles to the robotic hand.
In order for the robot to completely do the motion along with the desired trajectory q d and to overcome the existence of dynamic uncertainties, the experimental work was conducted to compare the effect of the unknown payload under NN learning with that without NN learning. The results illustrate that the NN-learning applied teleoperation system achieves the control of the DLR-HIT II robot hand with the smallest number of errors.