Automatically Select the Training Loss based on the Data Distribution for Talking Head Synthesis

Talking Head Synthesis is the study of how talking head video corresponds to audio speech. One of the challenges in talking head synthesis is how to balance the trade-off between model complexity and inference speed for real-world applications. In this paper, we focused on audio-to-lip movement prediction, which is the core of talking head synthesis. We developed an algorithm to automatically decide whether to use the regression-based training method, which is less general but more lightweight, to approximate the one-to-many mapping between audio and lip movement. The core idea behind the proposed approach is that the variation of paralinguistic features is strongly correlated with the variation of lip movement conditioned on the phonemes. When the variation of paralinguistic features is small, the variation of lip movement is also small, and in this case, regression-based methods can be used for lip movement prediction. We collected two Chinese-speaking datasets with different speaking styles and conducted experiments on them. The experimental results showed that the average audio-lip synchronization score and the real-time rate of the proposed algorithm were 3.8 and 1.2, respectively, outperforming the baseline, which was 3.55 and 1.6. The experimental results demonstrated that the proposed algorithm is effective for automatically balancing the trade-off between model complexity and inference speed based on the distribution of the training data.