Real-time pose estimation of an object manipulated by multi-fingered hand using 3D stereo vision and tactile sensing

We describe a real-time model-based sensing system to estimate the pose of an object manipulated by a multifingered hand in a realistic environment. The system uses the least-squares method to estimate the pose, on the assumption that the approximate initial pose and geometric shapes of the object and fingertips are known. 3D measurement of features on the surface of the object is achieved by visual tracking with stereo camera using a template matching method. The primary features used for visual tracking are distinguishable texture patterns on the object surface. In complex environments, visual tracking can be failed due to the various occlusion. The system solves the occlusion problem by detecting the occlusion and using contact information of fingertips with the 3D visual information. Actual implementation of the system is explained together with results of experiments about the stability of tracking, the accuracy of estimation, and the processing time of the system.