3D natural hand interaction for AR applications

In this paper we describe a computer vision-based 3D hand tracking system for a multimodal augmented reality (AR) interface. We have developed a 3D vision-based natural hand interaction method. This consists of four steps: (1) skin colour segmentation, (2) feature point finding, (3) hand direction calculation, and (4) simple collision detection based on a short finger ray for interaction between the userpsilas hand and augmented objects. The resulting fingertip tracking accuracy varied from 3 mm to 20 mm depending on the distance between the userpsilas hand and the stereo camera. We describe how this hand tracking is applied in three AR applications which merge gesture and speech input.