DreamPBR: Text-driven High-Resolution SVBRDF Generation with Multimodal Guidance

Existing material creation methods are limited in diversity due to the scarcity of real-world data. To enhance controllability and diversity, we propose DreamPBR, a diffusion-based generative framework that creates spatially varying appearance properties guided by text and multimodal controls. By integrating large-scale vision-language models trained on billions of text-image pairs with material priors from hundreds of Physically Based Rendering (PBR) samples, we achieve high-quality PBR material generation. We employ a material Latent Diffusion Model (m-LDM) to map albedo maps to latent space, which is then decoded into full Spatially Varying Bidirectional Reflectance Distribution Function (SVBRDF) parameter maps via a rendering-aware PBR decoder. To achieve diverse control, we introduce a multimodal guidance module that includes image and 3D shape guidance. We demonstrate DreamPBR’s effectiveness in material creation, showcasing its versatility and user-friendliness across various controllable generation and editing applications.