Multi-scale Feature Relation Modeling for Facial Expression Restoration

Facial expression analysis in the wild is easily vulnerable to the quality of facial images, such as low resolution or occlusion. Existing facial image restoration studies have mostly failed to take full advantage of facial expression prior information, which leads to loss of information related to facial expression in the restoration results. In this paper, we propose a multi-scale feature relation modeling GAN (MFRM-GAN) for facial expression restoration by exploring the multi-scale property and relationship of facial action units. Based on the GAN model, the MFRM-GAN integrates the graph convolution network (GCN) for feature relation modeling and feature pyramid network (FPN) for multi-scale feature extraction. Extensive qualitative and quantitative experiments on BP4D and DISFA datasets demonstrate that our proposed MFRM-GAN (i) can conduct facial expression in-painting and facial image super-resolution jointly, (ii) can recover better facial expression details comparing with state-of-the-art method in both visual effect and AU detection task.