Optical flow based lip reading using non rectangular ROI and head motion reduction

The mouth and the lips are deformed by muscles, and such deformations appear on the surface of the skin. Optical flow can be estimated by these deformations. In this study, we investigate optical flow-based lip reading. In natural speech, we slightly move our head, with speech without head movement being restricted. However, traditional optical flow-based approaches have not considered head movement of the speaker. Moreover, movement of the skin varies with facial placement. However, traditional approaches only consider a rectangular region of interest (ROI). This study addresses previous problems and proposes a novel optical flow-based lip reading method that includes head motion and employs facial features based ROIs. We demonstrate that our method exhibits the highest performance among several other methods for lip reading.