Reliable tracking of human arm dynamics by multiple cue integration and constraint fusion

The use of hard gestures provides an attractive means of interacting naturally with a computer generated display. Using one or more video cameras, the hand movements can potentially be interpreted as meaningful gestures. One key problem in building such am interface without a restricted setup is the ability to localize and track the human arm robust in image sequences. This paper proposes a multiple cite-based localization scheme combined with a tracking framework to reliably track the human arm dynamics in unconstrained environments. The localization scheme integrates the multiple cues of motion, shape, and color for locating a set of key image features. Using constraint fusion, these features are tracked by a modified Extended Kalman Filter that exploits the articulated structure of the arm. We also propose an interaction scheme between tracking and localization for improving the estimation process while reducing the computational requirements. The performance of the frameworks is validated with the help of extensive experiments and simulations.