Toward Fast Convergence and Calibration-Free Visual Servoing Control: A New Image Based Uncalibrated Finite Time Control Scheme

In this paper, a visual servoing robotic control scheme of fast convergence and calibration-free is proposed. The objective of this paper is to achieve a better convergence around the equilibrium for uncal-ibrated visual servoing systems. Moreover, there are threefold unknown parameters, the visual kinematic parameters, the dynamic parameters and the parameters of the feature points, considered in the design of the proposed control scheme. In order to achieve the above control objective, the ﬁnite-time tracking controller and three adaptive laws are proposed. The adaptability to the unknown parameters is guaranteed by the online adaptive laws. The ﬁnite-time convergence is achieved by the continuously non-smooth fractional order function in the controller. The rigorously mathematic proof of stability is given by homogeneous theory and the Lyapunov function formalism. Three real-time experiments are conducted to demonstrate the practical performance of the proposed scheme.


I. INTRODUCTION
The past decades have witnessed the rapid development of visual servoing robotic control. As a combination of computer vision and robot control, once appeared, the visual servoing control has attracted extensive attention of researchers, which is regarded as one of the most promising techniques. The uncalibrated visual servoing control is free of tedious calibration, which is laborsaving and flexible to the different parameters. The visual servoing control technique can be divided into three classes: image-based (IB), position-based (PB) and homography-based (HB). Compared with PB and HB, the image-based visual servoing control method purely employs information of the image errors, which is helpful to high degree of calibration-free [1], [2]. Consequently, we will focus on the image-based uncalibrated visual servoing control method in this paper.
Generally speaking, the uncalibrated visual servoing control is expected to be configuration-independent, pose-independent and camera-independent. To explain the The associate editor coordinating the review of this manuscript and approving it for publication was Sara Dadras . three independences, we need to clarify the kinematics of visual servoing systems.
It is well known that the visual servoing robotic system is highly nonlinear and more complicated than other robotic systems. The eye-hand mapping is determined by three factors: the visual configuration, camera intrinsic parameters and extrinsic parameters. According to the positional relation between the camera and the manipulator, the visual servoing configuration can be classified into two kinds: eye-in-hand (EIH) and eye-to-hand (ETH). The EIH configuration means that the camera fixed at the end-effector of the manipulator observes the reference objects; the ETH configuration means that the camera observes the reference objects fixed at the end-effector of the manipulator. As for the visual servoing control systems of visual-configuration-independence, there is no need to change any control parameters when the configuration changes. Namely, it does not rely on the specific configuration and can easily adapt to both EIH and ETH. The camera extrinsic parameters formulate the visual kinematic relationship, i.e., the transformation from the camera frame to the reference frame. Note that the positions of reference feature points usually are regarded as the camera extrinsic VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ parameters. The camera-independence means that the design of controllers does not rely on camera optics parameters and any other camera can be alternatives. In order to cope with the above parametric uncertainties, great effort has been paid to this field and positive progress has been made. Various uncalibrated visual servoing (UVS) control techniques have been proposed. Some publications present the neural networking method [3]- [6] and the genetic algorithm [7], [8] to cope with the estimation of visual nonlinear mapping. These artificial intelligence schemes employ off-line data to train the network models to approximate the nonlinear mapping models and usually require a mass of the data for the training stage, which is time-consuming.
In some publications, the UVS controller uses the approximation of the analytic Jacobian and the parameter perturbation in this method is stabilized by the design of the robust controllers [9]- [14]. Besides, there is another method called adaptive Jacobian UVS control in this field [15]- [17], which uses the adaptive laws to update the unknown parameters online and then constructs the Jacobian matrix using the estimates above. The core idea of the aforementioned two IBUVS methods is to analytically derive the estimation of the Jacobian matrix with uncertain kinematic parameters. Their differences mainly lie in the knowledge of the parameters: the former one requires approximate parameters but the latter one has no restriction on them. In light of this, in order to remove as much as restriction of parameters, we use the adaptive Jacobian-based scheme in this paper. Generally speaking, the adaptive Jacobian scheme uses the Slotine-Li method to online update the values of parametric estimations along the negative gradient of image error projections. Under the frame of adaptive Jacobian IBUVS techniques, the deployment of a visual servoing control scheme could be more flexible than other UVS control.
In the past decade, the rapid development of the adaptive Jacobian IBUVS control has been achieved. The proposed scheme in [21] is free of calibration of camera intrinsic and extrinsic parameters. By considering the delay problem in the networked IBUVS systems as well as the problems of camera-independence and configuration-independence, the article [22] proposes a networked protocol to stabilize the aforementioned networked IBUVS robotic systems. In practical applications, it can be much more complicated. The first challenge is the depth parameter. Some control schemes [23], [24] achieves the objective of regulation/tracking in image-space with the constant depth. It is well known that a constant depth cannot be satisfied in many practical servoing tasks. To address this, the controller which is capable to cope with a time-varying depth is needed. The difficulty originates from its nonlinearity in the perspective projection projection matrix, which makes the depth hard to be decoupled. In order to solve the problem, another kind of UVS control scheme called depth-independent adaptive control [25]- [27] is proposed, which can online update the time-varying depth separately.
The second challenge comes from the image/visual velocity measurement. The image velocity obtained from the numerical differential usually is a main source of deterioration. The use of the velocity measurements therefore is expected to be avoided in the controller design. In [25], [27], the IBUVS controller is proposed without using the image velocity measurement. But the controller is the type of setpoint and cannot be employed in a tracking control task. In [28], although there is no image-space velocity used in the phase of the estimation of kinematic parameters, the controller involves the image velocity. The avoidance of velocity in the UVS controller design has been a hot topic in this field.
The third challenge is the efficiency of control. The IBUVS schemes in [25], [26], [29] realize asymptotic image-space trajectory tracking. However, these schemes do not guarantee the time of convergence, which means that the convergence cannot be achieved in finite time under the given criteria and the convergence of these control schemes may be quite timeconsuming. By setting a big control gain to the controller, the convergent time can be remarkably reduced but it may lead to the deterioration of control performance as image noise increases [21].
Toward a higher degree of calibration-free and better control quality, IBUVS control has to face the above three challenges. Motivated by these challenges, we now investigate the fast convergence of IBUVS which considers unknown visual kinematics, unknown robotic dynamics and a time-varying image depth of the reference features. In conclusion, the main contributions of this paper can be summarized as follows: (a) A new FTS IBUVS controller in the form of a Proportion Differentiation plus (PD+) scheme is first proposed under multiple parametric uncertainties. (b) The nonlinear dynamics of robots is taken into account and the Lyapunov function formalism and homogeneous theory are used to rigorously prove finite-time stability of the system. (c) The results of contrast experiments are given to reveal the outperformance of the proposed IBUVS scheme to its counterparts.

III. MODEL FORMULATION OF VISUAL SERVOING SYSTEMS
A. KINEMATICS OF VISUAL SERVOING SYSTEMS As mentioned above, there are two typical configurations: eye-in-hand and eye-to-hand (see Fig.1(a) and Fig.1(b).) In the two configurations, the feature points which are expressed in the image-plane can be formulated as y = [y 1 , . . . y n ] T , where y i denotes the ith feature point coordinates. The relationship between image positions and Cartersian positions of feature points can be expressed as where z i (t) ∈ R denotes the depth w.r.t. the camera frame; x c i ∈ R 3×1 denote the unknown Cartesian coordinates of the feature point; T 1 , T 2 ∈ R 1×3 denote the 1st and 2nd rows of camera intrisic parameters matrix respectively; ∈ R 3×3 denotes the intrinsic parameter matrix satisfying Depth where T 3 ∈ R 1×3 denote the 3rd row vector of the matrix . Differentiating (7) yieldṡ Then differentiate (5) and by (5) and (8), one getṡ where u i (t), v i (t) ∈ R denote the coordinates of U-axis and V-axis respectively. In ETH configuration, the homogeneous transformation T c e from end-effector frame to the camera frame satisfies T c e = T c b ·T b e . Due to the fact that the camera is fixed w.r.t. the world frame as well as the robot base frame, the extrinsic parameter matrix T c b is constant. Then the following holds where x e i is the position vector of feature point w.r.t. the end-effector frame. Combining Eq.(9) and Eq.(10) yields the overall differential kinematics: where q(t),q(t) ∈ R n×1 denote the joint position and velocity vectors respectively; R b e , P b e ∈ SO(3) denotes the revolution matrix and translation vector; the row vectors m T 1 , m T 2 , m T 3 ∈ R 1×3 denote the 1st, 2nd and 3rd rows of the matrix M c b ; the matrix M c b ∈ R 3×3 means the projection matrix from image plane to the base frame of robot where R c b is the revolution part of the extrinsic parameters matrix T c b . The subscript b means the base frame and the superscript c means the camera frame.
By rewriting (11), it yieldṡ where the matrix J i (q) ∈ R 2×n is called total Jacobian of the system; the matrix D i ∈ R 2×n is the depth-independent Jacobian matrix and satisfies From the above equation, it can be clearly seen that the depth-independent Jacobian matrix is free of depth parameters, which achieve the goal of decoupling depth parameters. Besides, by differentiating z i (t), one haṡ . The above analysis derives differential kinematics of ETH configuration. In the EIH configuration, the transformation between the camera frame and the base frame is T c b (t) ∈ SO(3), including the camera extrinsic parameter matrix T c e and the forward kinematic matrix. In this configuration, the camera is mounted on the end-effector so the pose matrix T c e is constant. Then the following holds where x b i ∈ R 3×1 is the position vector of the ith feature point w.r.t. the base frame. Substituting (16) into (9) yieldṡ where m T 1 , m T 2 , m T 3 are the 1st, 2nd and 3rd row vectors of the matrix M c e ∈ R 3×3 which satisfies Then the depth-independent Jacobian matrix is and /∂q. By the comparison of ETH and EIH, it is not hard to find that the formulations of D i and d T i are similar. It is not hard to unify them in a common form. Rewrite Eq.(5) as follows where T(t) is the manipulator forward kinematic transformation matrix; x i is the position vector of the feature points in Cartesian space;M ∈ R 3×4 is the perspective projection. In the different configurations of visual servoing systems, the kinematic models can be unified in the same form by using different matrices and vectors as shown in Tab.1. The depth parameter can be unified by wherem T 3 ∈ R 1×4 is the 3rd row vector of matrixM.m T 3 and m T 3 are aligned with the dimension of T and R, P respectively. Eq.(11) and Eq. (17) can be unified bẏ where The matrices R, P respectively are the revolution part and translation part of the forward kinematic transformation. The difinition of the matrix M is given in Tab.1 according to the specific configuration. Differentiating (21) yieldṡ where Eq.(20) - (25) can be regarded as a unified kinmatic model. For simplification, we take the single feature case as an example to analyze hereafter and omit the subscripts of D i , d T i and x i . To facilitate analysis, it is necessary to introduce the following useful properties: Property 1: For a vector η ∈ R n×1 , the product Dη can be linearly parameterized as follows is the unknown parameter vector determined by the products of unknown camera and feature position parameters; p 1 satisfies p 1 ≤ 36. Property 2: For a vector, η ∈ R n×1 , the product d T η can be linearly parameterized as follows where Y k,2 (q, η) ∈ R 1×p 1 is the regressor matrix which does not depend on unknown parameters θ k ∈ R p 1 ×1 ; p 1 satisfies p 1 ≤ 36. Property 3: The depth z can be written as a linear form of unknown parameters as follows where Y z (q) ∈ R 1×p 2 is the regressor matrix; θ z ∈ R p 2 ×1 represents unknown parameters corresponding to the products of unknown camera and feature position parameters; p 2 satisfies p 2 ≤ 13.

B. ROBOT DYNAMIC MODEL
The well-known robotic Euler-Lagrangian dynamic equation is where τ is the n × 1 torque of the joints of manipulators; H(q(t)) is the n × n positive-define and symmetric inertia matrix; g(q) is n × 1 the gravitational force.; C(q(t),q(t)) ∈ R n×n is a skew-symmetric matrix such that for any proper dimensional vector ψ ψ T C(q(t),q(t))ψ = 0 Besides, it is not hard to linearly parameterize the robotic dynamic Euler-Lagrangian equation as follows: where θ d ∈ R p 3 ×1 denotes the unknown parameter vector and Y d (q,q,ξ ,ξ ) ∈ R n×p 3 denotes the regressor matrix and p 3 denotes the number of unknown parameters θ d .

C. PROBLEM STATEMENT
Basing on the preliminaries and analyses above, we present the problem as follows. Consider that the bounded desired trajectory is (y d (t),ẏ d (t),ÿ d (t)) and the system is formulated by (20)∼ (30). There are unknown parameters inlcuding the camera instrinsic parameters , camera extrinsic parameters T c e (T c b ), the feature positions x and dynamic parameters in the system. The problem of this paper is to design a proper controller to guarantee the visual servoing robotic system converge to the desired trajectory in finite time.

IV. FINITE-TIME IBUVS CONTROLLER DESIGN
Define the image tracking errors by y = y − y d , ẏ = y −ẏ d , and construct the reference of image velocity bẏ where λ ∈ R is the constant to be determined. Basing on the depth-independent Jacobian D and the vector d, we define the compensated-depth-independent Jacobian matrix Q as where α 1 is the constant to be determined. As mentioned by Property 1 and Property 2,D andd T can be online estimated by the vectorθ k which will be presented later. According to reference image-velocityẏ r , we define the reference joint velocityq r bẏ q r =ẑQ +ẏ r , whereẑ denotes the estimated depth andQ + denotes the pseudo inverse, namely, Here, it is assumed that the Jacobian matrixQ + will not lead to the singularity ofQ. By the reference velocityq r , one may construct the new sliding vector as follows Now, we are at the position to propose the controller of this paper: where K s ∈ R n×n and K y ∈ R 2×2 are the gain matrices to be determined later, and 0 < α 1 < 1, α 2 = 2α 1 /(1+α 1 ). α 1 and α 2 are the degree orders of sig(s q ) and sig( y) respectively; Y d (q,q,q r ,q r )θ d is the esimation of the robotic dynamics, given by whereθ d denotes the estimating vector of the unknown dynamic parameters. VOLUME 8, 2020 The controller (36) is designed by the typical form of the proportional differential plus forwarding dynamics compensation of robotic systems. The first term of (36) is the compensation of forward dynamics compensation, which is online estimated by the adaptive law of unknown dynamic parameters. The compensation of dynamics is beneficial to eliminate the negative effects of the nonlinearity of manipulators in high speed movement and the varying workload. The second term is the proportional term of the image errors with fractional order. The term provides a time-varying control input to force the states of systems to the equilibrium. The last term includes the velocity state in joint-space and is for the purpose of improvement of fast response of the systems. The control block diagram of the proposed controller is shown in Fig.2.
The signum function in Eq.(36) is defined by where ξ = [ξ 1 , . . . , ξ n ] T ∈ R n , 0 < α < 1, and sgn(·) is the standard signum function. Moreover, sig(ξ ) satisfies According to Property 1 and Property 2, the estimationQ can be linearly parameterized bŷ and the estimated error is Likewise, according to Property 3, the estimated depthẑ is given byẑẏ and the estimated error of the depth can be derived from In the adaptive Jacobian IBUVS scheme, the unknown parameters θ k , θ z , θ d are updated by the updating laws. To this end, we present the following three lawṡ where d , k and z are the gain matrices to be determined later. Define x 1 := y, x 2 := s q , 5 ] T . Based on the above analysis, the closed-loop equation can be formulated by 88338 VOLUME 8, 2020

A. STABILITY ANALYSIS
The main result of this paper can be summarized by the following theorem. Theorem 1: The controller (36) and the adaptive laws (44) -(46) guarantee the convergence of the image-space tracking errors and the estimated errors, i.e., lim t→∞ y, ẏ = 0, by the positive constants λ and the positive definite symmetry matrices K s ∈ R n×n , K y ∈ R 2×2 ; d , k and z with proper dimensions, 0 < α 1 < 1, α 2 = 2α 1 /(1 + α 1 ). Proof: The proof of this theorem can be divided into two steps: 1) The asymptotic stability of the closed-loop system; 2) The locally finite-time stability at the equilibrium. Then the global finite-time stability of the system can be directly derived by Lemma 1.
In conclusion, by Barbalat's Lemma and Lemma 2, it is not hard to derive lim t→∞ y, ẏ = 0.
Remark 1: In this paper, the parameter identification θ k → 0, θ z → 0, θ d → 0 is not the goal of the proposed method. The control objective is to guarantee the convergence of the tracking errors of the image velocity and position. Namely, if Y k (y, y d ,q,q) θ k = 0, Y z (y, y d ,ẏ d , q) θ z = 0, and Y d (q,q,q r ,q r ) θ d = 0 are guaranteed, the convergence of close-loop systems can be achieved. On the other hand, the elements of unknown parameters vectors are not corresponding to actual kinematic and dynamic parameters but the new constants related to kinematics and dynamics. Identifying them is not our concern.

V. EXPERIMENTS
To verify the effectiveness of the proposed IBUVS control method, the following experiments are conducted.

A. EXPERIMENTAL SET-UP
The experimental hardware platform is given in Table 2. As aforementioned, both the EIH and ETH configurations are constructed by two different cameras fixed on the end-effector and the tripod respectively. The cameras offer 30FPS video stream and high DPI to servoing control thë  Under the EIH configuration shown as Fig.3(a), the Logitech camera C310 is mounted on the end-effector. In the experiment, there are four different feature points on the color board shown in Fig.3 and

B. EXPERIMENTAL RESULTS
To show the performance of the proposed finite-time adaptive controller, the experimental goals are: the adaption to uncertainties and the rapid convergence of the proposed controller.
Firstly, in order to verify the effectiveness of noncalibration, both EIH and ETH are considered as well as the   In the EIH configuration, the color board is placed at the three poses as shown in Fig.5 The camera C310 is fixed on the end-effector of the manipulator and its actural pose is In order to verify the adaption to the unknown extrinsic parameters, the following two poses of the color board are considered   The first experiment is conducted by placing the color board at the above three different poses and the experimental results are given in Fig.5 and Fig.8.  From Fig.5, it can be seen that the manipulator presents two different circles with POSE-1 and POSE-2 of the color board in Cartesian space. It means that the proposed IBUVS-F control scheme can ensure the UVS adaptive to the different poses of the reference feature points. Namely, it guarantees the manipulator track the desired trajectory in image-space with different poses. From Fig.6, one may easily observe that the image trajectories of the four features, starting from the initial positions, converge to the desired circles and demonstrate a good performance of tracking with POSE-1 and POSE-3. The profile of convergence with each pose can be seen clearly in Fig.7. Besides, Fig.8 shows that the joint velocities of the manipulator in POSE-3, which can be seen that the controlled system can rapidly respond with low noise.
The results of the first experiment verify the adaption to the poses of the reference feature points. In the second experiment, the adaptions to the different instrinsic parameters and to the extrinsic parameters will be verified. To achieve the goal, the visual configuration is changed to ETH.
As shown in Fig.4, in the ETH configuration, the reference frames of the cameras are at two different poses:   The pose from the color board to the end-effector is The second experiment is conducted by placing the camera at the above two different poses and the experimental results are given in Fig.9 and Fig.11. From Fig.9, it can be seen that the manipulator presents two different circles with the camera pose 1 and the camera pose 2 in Cartesian space. It means that the proposed IBUVS-F control scheme can ensure the UVS adaptive to the different poses of the camera. Besides, the camera is different from the one in EIH configuration. Hence, it means that the control scheme is adaptive to the intrinsic parameters. From  Fig.10, one may easily observe that the image-space trajectories of the four features also demonstrate a good performance of tracking with POSE-1 and POSE-2 of the camera. The profile of convergence with each camera pose can be seen clearly in Fig.11.
In the above two experiments, the proposed IBUVS scheme guarantees the successes of visual servoing tasks even with a large difference of camera intrinsic and extrinsic parameters. The effectiveness of non-calibration is verified.
In order to verify the fast convergence of the proposed IBUVS, the third experiment is conducted. The proposed finite time IBUVS, called IBUVS-F, is compared with other two reference control schemes in this experiment. One is the adaptive control scheme of asymptotic stability (IBUVS-A) in [31] and the other is an adaptive gain scheduling control method (IBUVS-AAG) in [32]. The adaptive gain in IBUVS-AAG satisfies: where x = e ∞ (x = e 2 ) with e denotes the image position errors; a = λ(0) − λ(∞), b =λ(0)/a, c = λ(∞), λ(0) denote the gains at x = 0; λ(∞) denotes the gain at x = ∞.
To quantitatively evaluate the performance of the convergence, the index of the convergent time is employed. The convergence is determined by if the average pixel errors of four features are lower than 10. Seven test groups among IBUVS-A, IBUVS-AAG and IBUVS-F are conducted and the experimental results are given in Tab.5 and Fig.12.
From the above comparisons, it can be easily observed that when the large control gains are used, the differences of the three schemes are small; but they become much larger than before when the control gains decrease. Moreover, the convergent time of IBUVS-F is remarkably less than the  others in group 3-7, which demonstrates the improvement of the performance in the convergence.
One may note that the effect of control gains is a key factor of time of convergence, from Tab.4 and Fig.13. But it does not mean that the big gains are superior to the small ones. In fact, however, the case with a small gain gives the good traits of a smooth control. Specifically, it is beneficial to the uncalibrated visual servoing systems being more reliable and stable than the one with big gains. Also, it facilitates the systems to have a smooth action trajectory of manipulators in Cartesian space. In the proposed scheme, when the error are about to zero, the IBUVS-F scheme with the fractional orders offer a larger effort to stabilize the system than the asymptotic ones. It also demonstrates the superiority of the proposed method. Hence, without using big control gains, the improvement of the rate of convergence around the equilibrium is of significance.

VI. CONCLUSION
In this paper, the finite-time visual seroving control for image-based uncalibrated visual servoing systems is proposed. In the field of IBUVS control, the method for the fast trajectory tracking by using finite-time control theory is originally proposed to address the problem of slow convergence at the equilibrium. The novelties of the proposed method lie in the following aspects. a) Threefold unknown parameters are simultaneously considered in this paper, while the existing work usually considers single or twofold uncertainties of uncalibrated visual servoing systems. b) The homogeneous theory is firstly employed to resolve the finite-time stability of UVS systems as far as the authors know.
With the proposed calibration-free IBUVS control scheme, either EIH or ETH configuration can be randomly used. As well as the intrinsic and extrinsic parameters of the camera can be unknown. By using the homogeneous theory and Lyapunov function formalism, the stability of the closed-loop system is rigorously proven. Three realtime experiments are conducted to verify the effectiveness of IBUVS-F from the following two aspects: the adaption to uncertainties and the rapid convergence of the proposed controller. The adaption of IBUVS-F is verified by the EIH and ETH configurations and by two different cameras at the different poses. The fast convergence is verified by the contrast experiments among IBUVS-F and the other two existing control schemes. The experimental results demonstrate the validation of the proposed method. VOLUME 8, 2020 There is no denying that the performance of the proposed IBUVS-F is limited under the noisy image space due to the rapid response to image errors. A possible solution to remove the limitation is by using an image observer to eliminate the negative effects of image errors without improving the performance of visual sensors. Therefore, it is an interesting and promising topic to be investigated in the future.
YONGHENG WANG received the B.S. degree in agricultural mechanization and its automation from Tianjin Agricultural University, Tianjin, China, in 2018, where he is currently pursuing the master's degree in agricultural engineering. His current research interests include intelligent agricultural robot technology and the application of mechatronics in precision agriculture.
KUN YOU received the B.S. degree in measurement and control technology and instrumentation from Tianjin Agricultural University, Tianjin, China, in 2018, where he is currently pursuing the master's degree in agricultural engineering. His current research interests include intelligent agricultural robot technology and the application of mechatronics in precision agriculture. VOLUME 8, 2020