Audio-Visual Speech Reconstruction for Communication Accessibility: A ConvLSTM Approach to Indonesian Lip-to-Speech Synthesis

Significant communication barriers created by severe voice disorders like aphonia motivate the development of new accessibility technologies. While previous Indonesian language research has focused solely on lip-to-text conversion, this study extends capabilities to direct lip-to-speech synthesis, representing a significant advancement in accessibility technology. This research addresses this accessibility challenge by developing an innovative machine learning model that synthesizes synchronized speech directly from lip movements, providing a non-invasive communication alternative for individuals with voice loss. The proposed system processes audio-visual data through Mel spectrograms to create an optimized latent space representation of (64, 15) dimensions, utilizing the Lumina dataset—a specialized corpus of Indonesian lip movements recorded under controlled conditions across 14 diverse speakers. The methodology incorporates a two-stage sequential training approach combining an audio speech model and a lip-to-speech model. Advanced deep learning architectures including 3D and 2D Convolutional Layers alongside Convolutional Long Short-Term Memory (ConvLSTM) networks effectively capture both spatial and temporal patterns of lip movement. Transfer learning with EfficientNetV2 enhances feature extraction while reducing computational requirements. Comprehensive evaluation using industry-standard metrics demonstrates promising results, with maximum values of 1.604 for PESQ, 0.7217 for STOI, and 0.5102 for ESTOI. These findings highlight the substantial potential of deep learning in overcoming communication barriers for Indonesian speakers.