Toward Language-independent Lip Reading: A Transfer Learning Approach

Automated lip-reading, i.e., translating lip movements into text, has received growing interest in recent years with the success of deep learning across a wide variety of tasks. One major obstacle to progress in this field has been the lack of suitable training resources, with the vast majority being limited to a selective set of languages. In this paper, we study the effectiveness of transfer learning to address the lack of massive amounts of labeled data for building a language-independent lip-reading system. Towards this target, we exploit existing knowledge and generalize to new languages via deep neural networks. Experimental validation is carried out on several publicly available data, i.e., LRW for English, LRRo for Romanian, and LRW-1000 for Mandarin showing promising results with significant performance improvements of the multilingual models in contrast to the monolingual ones.