Development of Sensory-Motor Fusion-Based Manipulation and Grasping Control for a Robotic Hand-Eye System

In this paper, a sensory-motor fusion-based manipulation and grasping control strategy has been developed for a robotic hand-eye system. The proposed hierarchical control architecture has three modules: 1) vision servoing; 2) surface electromyography (sEMG)-based movement recognition; and 3) hybrid force and motion optimization for manipulation and grasping. A stereo camera is used to obtain the 3-D point cloud of a target object and provides the desired operational position. The AdaBoost-based motion recognition is employed to discriminate different movements based on sEMG of human upper limbs. The operational space motion planning for bionic arm and force planning for multifingered robotic hand can be both transformed as a convex optimization problem with various constraints. A neural dynamics optimization solution is proposed and implemented online. The proposed formulation can achieve a substantial reduction of computational load. The actual implementation includes a bionic arm with dextrous hand, high-speed active vision, and an EMG sensors. A series of manipulation tasks consisting of tracking/recogniting/grasping of an object are implemented, and experiment results exhibit the responsiveness and flexibility of the proposed sensory motion fusion approach.