A new measurement methodology of multi-finger tracking for handheld device control using mixed reality

This paper proposes a new method to measure a hand and its multi-finger positions when a user is manipulating button controls of a handheld device. We developed SPIDAR-Hand(Space Interface Device for Artificial Reality simulating Handheld Device Usability) for tracking each fingertip positions and giving force feedback to each finger tip by combining AR technique for pinking up the hand movement. When using a handheld device, user commonly press a button by his or her thumb while grasping the device body with other fingers. The SPIDAR-Hand calculates a 3D position of each fingertip besides thumb using the lengths of three strings and the thumb with four strings in order to improve the accuracy. We introduce an AR technique for monitoring the hand movement by tracing the hand in motion images. More specifically the image recognition technique assists SPIDAR-Hand as it can not solely give precise hand position and rotation due to the difference in width and length of user's hand factors that would be important when designing more comfortable control for handheld devices.