Low-Latency Real-Time Audio-Driven Talking Head Generation Based on Future Speech Feature Prediction

This research proposes a low-latency real-time audio-driven talking head generation (THG) method based on future speech feature prediction. Most traditional methods often suffer from high latency, hindering real-time interaction because they inevitably require future audio context to maintain generation quality. To address this, we predict future speech frames from mel spectrogram features, enabling the generation of facial animations in advance and reducing latency. Our approach integrates the Neural Radiance Field (NeRF) model for high-fidelity facial animation, ensuring precise synchronization with speech input. Extensive experiments demonstrate the high prediction accuracy of the proposed future speech feature prediction module. Furthermore, comparisons with previous works confirm that our method significantly reduces latency while improving video quality. The key contribution of this work is the introduction of future speech feature prediction to real-time THG, effectively reducing synchronization latency while maintaining high video quality.