Motion-based finger tracking for user interaction with mobile devices

A new motion-based tracking algorithm for user interaction with hand-held mobile devices is presented. The idea is to allow mobile phone users to control the device simply by moving a finger in front of a camera. A novel combination of Kalman filtering and the expectation maximization (EM) algorithm is utilized for estimation of two distinct motion components corresponding to the camera motion and the finger motion. The estimation is based on motion features, which are effectively extracted from the scene for each image frame. The performance of the technique is evaluated in experiments which show the usefulness of the approach. The method can be applied also when some conventional finger tracking techniques such as color segmentation and background subtraction cannot be used.