Physics-Based Appearance and Illumination Estimation from a Single Face Image

Deep learning-based 3D face reconstruction methods have shown promising results in facial shape estimation. However, due to the intrinsic ill-posedness, few of them succeeded in recovering high-quality facial appearance and lighting conditions, which largely limits the re-rendering performance. In this paper, we propose a novel physics-based deep learning framework for human facial appearance and illumination estimation. A computation-friendly physics-based model is proposed to simulate complex real-world illumination effects, e.g., specularities and cast shadows. Combining with the proposed Normal Fusion Block (NFBlk) and online illumination augmentation scheme, our framework is capable of estimating dense illumination and full-posed appearance from a single face image. Experiments demonstrate that, even with limited amount of training data, the proposed framework can achieve promising results in different face manipulation tasks, such as face relighting, reenactment and swapping.