Robust recognition of faces and facial features with a multi-modal system

We use a combination of shape and texture analysis, color segmentation and motion information for finding the positions of whole faces plus the precise location and shape of the mouth. Combining several modalities improves the robustness of the analysis considerably and allows handling of a wide variety of conditions. Mouth shapes can enhance the accuracy of speech recognition systems, in particular under noisy conditions. However, finding the shape of the mouth precisely under unrestricted conditions is a challenging task. To be of practical value a system must handle different complexions of people as well as variations in lighting, different head orientations, moustaches, beards, and glasses. To deal with such a diversity of conditions, our system includes several different models of the face and the mouth area. New faces are compared to these models and the most representative one is chosen for the analysis. We tested our system on samples from video sequences of 50 different speakers. When trained on a particular person, the mouth location is found correctly in more than 98% of the images. When trained on a random set of 10 people from the database, the system handles typically 87% of the other people correctly. In speaker-dependent lip reading experiments we observed 93% word accuracy on five-digit strings.