High-Fidelity Texture Generation for 3D Avatar Based On the Diffusion Model

Restoration of texture plays a pivotal role in 3D facial reconstruction tasks. In recent years, generative models have been used widely in 3D facial reconstruction for generating textures. Nonetheless, textures produced by these methods are frequently affected by the lighting coefficients predicted by the 3D morphable models or exhibit non-uniform illumination, making it difficult to adapt to diverse scenarios. Furthermore, it sometimes lacks realistic facial intricacies, thereby impeding the attainment of high fidelity outcomes. Recently, diffusion models have demonstrated impressive performance in the realm of image generation. This paper presents a UV-texture generation approach grounded in diffusion models. It integrates a pre-trained conditional diffusion model as a texture generator into a 3D morphable model, simultaneously optimizing two sets of input parameters of the generator. The model is trained on a UV-texture dataset featuring uniform illumination. Leveraging the diffusion model's capability to reconstruct images beyond the training distribution, it effectively models UV texture maps exhibiting both uniform illumination and high fidelity simultaneously. Additionally, we further elucidate the generation process and propose a method for texture color enhancement and customization. The method can restore realistic texture maps containing environmental lighting information and even other custom colors.