A robust hierarchical lip tracking approach for lipreading and audio visual speech recognition

This paper presents a robust hierarchical lip tracking approach (RoHiLTA) for lip-reading and audio visual speech recognition (AVSR) applications. Lip regions of interest are subtly detected by motion and facial structure information. Improvements are made on active shape models (ASMs) for extracting lip contours more accurately and efficiently from video sequences of a speaker's talking face in natural lighting conditions and without particular make-ups. Local and global ASM search algorithms are both improved by introducing color information, 2D mouth corner match, and robust estimation. For noise-free features, localization errors are automatically corrected by an interpolating scheme. A fast implementation of the hierarchical approach is also proposed. Extensive experiments show that the improved ASM can effectively reduce the lip locating errors. The fast implementation of RoHiLTA can consistently achieve superior performance to conventional ASMs in lip tracking tasks, and then can be effectively integrated in lip-reading and AVSR systems.