3D Face Reconstruction with Graph Attention Based on Action Unit Features

The reconstruction of 3D face shapes and expressions from 2D images remains unconquered due to the lack of detailed modeling of human facial movement based on the correlation between the different parts of faces. We propose to use facial action units (AUs), which are detailed taxonomy of the human facial movements based on observation of activation of muscles or muscle groups, in the 3D facial expression reconstruction to allow for detailed modeling of various facial expression types. We present a novel 3D face reconstruction framework called Action Unit feature-based Graph Attention Network Encoder (AUGANE) that can generate a 3D face model that is responsive to AU activation given a single monocular 2D image to capture expressions. AUGANE leverages AU-specific features as well as facial global features to enable accurate 3D facial expression reconstruction with Graph Attention Network Encoder (GANE). We also introduce a novel loss function which is to force learning toward the minimal discrepancy in AU activation between the input and rendered reconstruction. The experimental results demonstrate the superior performance of the proposed framework over state-of-the-art methods.