Motion segmentation and appearance change detection based 2D hand tracking

In this paper a novel method called MACS for 2D hand tracking using motion and color information for head-worn monocular color cameras is presented. Many head-mounted devices (HMDs), like eye trackers or Augmented Reality glasses, are equipped with only one color camera capturing the field of view of the user from the ego perspective. To interact with the HMD, hand gestures are an intuitive modality. Hand tracking can be viewed as the first step towards hand gesture recognition. But to recognize hand gestures in industrial or commercial applications, the hand tracking process must produce robust estimations in each frame, given any lighting conditions inside and outside of a building. To achieve this, the presented method creates a motion segmentation, determines a foreground segment and tracks the color appearance of this segment over time to estimate robustly when and roughly where the hand is visible. A sophisticated skin color detection method, fused with the previously generated moving foreground segment, makes the estimation of the hand trajectory more accurate using a simple particle filter with a specialized observation model. This allows for estimating the 2D hand position even in front of complex backgrounds and difficult lighting conditions. A comparison of this new algorithm with other tracking methods is conducted using a thorough evaluation methodology and challenging image sequences with more than 25,000 frames containing different wiping hand gestures.