IF-GAN: Generative Adversarial Network for Identity Preserving Facial Image Inpainting and Frontalization

Face recognition has seen rapid development and widespread usage in recent years. However, recognizing faces with occlusions or in large pose variations still poses great challenges for existing face recognition algorithms. While these two obstructions frequently occur in real-world applications simultaneously, many recent research efforts can only solve one of these two problems. To address this issue, in this paper, we propose an identity preserving two-stage generative adversarial network that can simultaneously complete the task of de-occlusion and face frontalization. Unlike previous works that can only inpaint synthetic rectangular occlusion which is unlikely to occur in real-life scenarios, we employ partial convolutions in our face inpainting stage to handle the realistic irregular occlusions. For face frontalization, we use a dualpathway structure that processes global facial shape and local facial fiducial region details separately, allowing the network to gain enough semantic information to faithfully reconstruct the frontal facial image. Both stages are supervised on both pixel and feature levels such that the network can produce photo-realistic yet identity preserving unoccluded frontal facial images. Qualitative and quantitative experiments demonstrate that the proposed approach is able to improve the performance of existing face recognition systems by restoring identity information contaminated by occlusions and pose variations.