Action Unit Based Smiling Face Generation Method

The widespread usage of facial expression processing in digital entertainment and data augmentation has drawn rising attention. However, the existing works on facial expression generation still fall short in reality and continuity, especially on smiling faces, which is not natural enough. In this paper, we propose a smiling face generation method based on Action Unit (AU) to address the aforementioned problems. Our method mainly adjusts the magnitude of target AU according to the features of smiling faces and the person itself to achieve better generation effect. We propose two ways to adjust AU, called average AU and relative AU respectively. Average AU reflects the AU configuration with universal smiling face expression and relative AU combines the AUs represent smiling expression with other AU of source image. We collect smiling face images to accumulate new datasets where we find the intrinsic pattern of AU. We leverage average AU and relative AU to optimize the process of smiling faces generation and the experiment results indicate that our method can achieve better facial expression manipulation compared with the baseline models. In addition, we conduct user evaluation experiment from which we get following results: 1) the results of L2 distance and Fréchet Inception Distance (FID) indicate that degree of the facial expression realization of the images generated by our method is improved greatly; 2) the results of the questionnaire indicate that the images generated by our method win more human visual preferences and are of higher quality.