Realistic and Stable 3D Gaussian Relighting: Semantic-Guided Stable Material Estimation through Object-Oriented Differentiable Gaussian Path Tracing

The Gaussian relighting is critical for appearance editing and physical-based rendering, which rely on the decomposition and estimation of light and materials. However, existing methods treat both the environmental lighting and material features as unknowns and estimate them jointly, which results in inaccurate material decomposition and estimation due to the inherent variation of environmental illumination across datasets. In this study, we introduce the realistic and stable 3D Gaussian relighting (RS3DGR) that is enabled by the semantic-guided stable material estimation through object-oriented differentiable Gaussian path tracing. First, the target object is segmented from environment via semantic 3D Gaussian segmentation and the semantic label is designed as the roughness regularization term in optimization. Then, the object is illuminated by the environmental light sampled from the radiance field of environmental Gaussians or other types of radiance caches. The object-oriented path tracing rendering is differentiable and efficient. Under the alpha blending of primary rays penetrating through object Gaussians, the secondary rays are uniformly emitted from those primary-hit Gaussians to directly sample the environmental radiance field to model the reflection on object surface. As a result, the influence of unstable environmental lighting is removed, and the semantic prior successfully promotes the convergence on material estimation. Experimental results demonstrate significant improvements in stability and accuracy for material reconstruction on both synthetic and self-constructed datasets.