HumanRef-GS: Image-to-3D Human Generation With Reference-Guided Diffusion and 3D Gaussian Splatting

Generating a 3D human model from a single reference image is a challenging task as it involves inferring textures and geometries in unseen views while maintaining consistency with the reference image. Existing methods that rely on 3D generative models are limited by the availability of 3D training data. Optimization-based approaches that distill text-to-image diffusion models into 3D models often struggle to preserve the intricate texture details of the reference image, resulting in inconsistent appearances across different views. In this paper, we propose HumanRef-GS, a novel method for single image-to-3D clothed human generation based on 3D Gaussian Splatting (3DGS). To ensure the generated 3D model is both photorealistic and consistent with the input image, HumanRef-GS employs a unique technique called reference-guided score distillation sampling (Ref-SDS). This method effectively incorporates image guidance into the generation process, enhancing the quality of the results. Additionally, we introduce region-aware attention to Ref-SDS, which ensures accurate correspondence between different body regions. To mitigate the impact of view dependence in 3DGS and enhance the view-consistency of the generated results, we substitute the anisotropic Gaussians in the vanilla representation with isotropic Gaussians. By utilizing the 3D Gaussian representation, our method significantly enhances the generation efficiency and rendering speed of 3D clothed human models. This improvement allows for faster and more efficient generation of high-quality results. Experimental results demonstrate that HumanRef-GS surpasses state-of-the-art methods in generating 3D clothed humans with fine geometry, photorealistic textures, and view-consistent appearances. We are committed to making our code and model available upon acceptance for further research and exploration.