3D Hand Tracking With Head Mounted Gaze-Directed Camera

This paper investigates hand tracking for everyday manipulation tasks using a gaze-directed camera, which is a wearable camera that actively directs the visual attention focus of the person who wears it. The proposed 3-D hand tracking algorithm is implemented as a submodule in the integrated vision system, which tracks the positions and the poses of the acting hands, the pose that the manipulated object, and the pose of the observing camera at the same time. Such system provides comprehensive data for learning predictive models of vision-guided manipulation that include the objects people are attending, the interaction of attention and reaching/grasping, and the segmentation of reaching and grasping using visual attention as evidence. The key contribution in this paper is the hand tracking algorithm in the ego view based on the interaction between the 2-D and 3-D hand model. The algorithm uses 2-D model tracking results to initialize and predict 3-D tracking, which reduces the number of particles and makes it possible for pose estimation by the proposed multilayer hierarchical sampling, which is the key techniques for estimating the state in high dimensional state space. Moreover, by incorporating the tracking result of the manipulated object, the proposed algorithm is capable of estimating the pose and position of the hand despite substantial occlusions caused by the manipulated object. We validate the proposed hand tracking algorithm using the whole integrated vision system in the context of kitchen tasks.