Syllable-Based Indonesian Lip Reading Model

Lip reading is a communication method by reading the lips movement of a speaker. It is also called visual speech recognition, which converts a video into a text. The text is consisting of some words or even sentences spoken by the speakers. One of the challenges often encountered in a lip reading is the high variances of inputs. The variances, like facial features and different speed of speech, can decrease the accuracy. Nowadays, deep learning provides promising results in extracting visual features. In order to be able to use a video as the input, a 3D Deep Learning architecture is exploited. Besides, the out-of-vocabulary (OOV) problem also makes the visual speech recognition system harder to apply in the real world. It can only predict the words appear in the dictionary. However, the vocabulary continues to grow each year, especially in the Indonesian language. It is hard to fit all possible words into the system. Hence, a syllable-based model is proposed in this research to handle such a problem. The syllable-based model gives a chance to build a new word that does not appear in the dictionary. The combination of the existing syllable is used to construct a new word. Since the data obtained too small for deep learning, the augmentation process is performed 40 times. Evaluation using the augmented data, the proposed model reaches a high accuracy of 100% for the testing set. An examination using ten OOV words informs that the developed model gives a lower accuracy of 80%.