A virtual mirror interface using real-time robust face tracking

We describe a virtual mirror interface which can react to people using robust, real-time face tracking. Our display can directly combine a user's face with various graphical effects, performed only on the face region in the image. We have demonstrated our system in crowded environments with open and moving backgrounds. Robust performance is achieved using multi-modal integration, combining stereo, color and grey-scale pattern matching modules into a single real-time system. Stereo processing is used to isolate the figure of a user from other objects and people in the background. Skin-hue classification identifies and tracks likely body parts within the foreground region. Face pattern detection discriminates and localizes the face within the tracked body parts. We show an initial application of the mirror where the user sees his or her face distorted into various comic poses. Qualitatively, users of the system felt the display "knew" where their face was, and provided entertaining imagery. We discuss the failure modes of the individual components, and quantitatively analyze the face localization performance of the complete system with thousands of users in recent trials.