High-Quality Talking Face Generation via Cross-Attention Transformer

Lip-sync is the task of synchronizing the lip movements of a speaking person with the corresponding audio. It is a critical component of many multimedia applications. Achieving high-quality lip-sync requires accurately modeling the relationship between audio and visual information, as well as the details of human face. To date, existing talking face generation methods can be classified into two types. One type uses intermediate representations such as landmarks and 3D models to generate lip movements which typically result in a stiff look and unnatural lip movements. The other type of method directly generates images by leveraging the GAN-based generative model, resulting in low resolution, missing teeth, and instability in the training process. To overcome these limitations, we propose a two-stage method which utilizes shift-window based cross-attention mechanism and a vector-quantized Generative Adversarial Network (VQGAN). In the first stage, we use VQGAN to reconstruct fine-detailed images and obtain a latent codebook of the talking face image. In the second stage, we introduce a newly designed spatial shift window attention model to generate a latent representation of the synchronized image conditioned on audio input. Our model effectively utilizes the sparsity of attention for video generation. The direct application of a conventional Transformer in this context results in a protracted, iterative procedure that is suboptimal for real-time performance. Conversely, our adapted Transformer network leverages these distinct characteristics, facilitating the generation of output in a singular step. This significantly enhances the network’s real-time computational efficiency while maintaining its performance integrity.