An investigation on the usage of image quality assessment in visual speech recognition

Having a robust speech recognition scheme that can be relied upon in different environments is a strong requirement for modern systems. Previous works in field of lipreading mainly have used a level of segmentation at the beginning and then used the structure of the mouth, facial muscles of the speaker, some critical points on the lip, or the motion of these points for word recognition. In this paper we present a novel way of processing the video signal for lipreading application. We neither used segmentation level nor the extraction of important facial points. Instead, we've used HVS (human visual system) based image quality metrics, especially complex wavelet structural similarity (CW-SSIM) and visual information fidelity (VIF) as our similarity criterions. We used an intelligent frame by frame video comparison technique and we applied mentioned metrics in our approach. Experimental results showed that in comparison to other methods, this novel method can recognize the true letter among the letters of the utilized dictionary with an acceptable accuracy.