Phoneme-based lipreading of silent sentences

With the development of computer vision algorithms, more and more researches on lipreading have been conducted. Most of them use lipreading as support to speech recognition, and based audible videos to predict words. But complex and long sentences can easily lead to semantic ambiguity and the recognition process relies too much on audio information. In this paper, we designed a special method for lipreading based on phonemes that can predict long sentences without audio information. First, we divide the lipreading task into two processes, phonemes are used as taxonomic units, identify phonemes from images, and then predict sentences from phonemes. Second, we use a specially designed spatiotemporal convolution called TCN to classify phonemes, and informer to convert phoneme sequences into sentence sequences, because the introduction of TCN module helps to extract the contextual information of sentences better, thus reducing the probability of ambiguous recognition. The experiments are compared and analyzed with other lipreading methods, and the results show that our model performs better on GRID, LRS, and LRS2 datasets.