VAR: An Efficient Silent Video to Speech System with Subtitle Generation using Visual Audio Recall

In this paper, a method to generate speech from a muted video is proposed. Recent advances in computer vision and deep learning have allowed the development of lip-reading technology that can recognize and interpret lip movements in videos. However, one of the main challenges in visual-to-speech synthesis is distinguishing between homophenes, or sets of words that look identical on a person's lips but contain different sounds. To address this challenge, the proposed model utilizes a speech generation framework that uses an external memory called Visual Audio Recall (VA-Recall) to generate speech from silent video. The VA-Recall system allows the system to effectively synthesize speech from a speaker by recalling the most relevant auditory information that matches the visual input context. The proposed method further generates subtitles for the reconstructed audio. Overall, this process allows for a more natural and contextually appropriate speech synthesis system.