Creating and Experiencin 3D Immersion Using Generative 2D Diffusion: An Integrated Framework

To address the challenges of lacking user engagement in current AI-based 3D generation developments, requires improved visual quality, accelerated workflow, and engaging interactivity. This paper presents a generative AI-powered framework for creating and experiencing personalized 3D environments through a coherent process. Our approach Leverages text-to-image diffusion models and is tailored for panoramic features and sophisticated visual aesthetics. Video diffusion models are employed for introducing dynamic elements. The results are transformed into spatial data through monocular depth estimation and integrated into the 3D model. The input module offers a playful interface allowing visually manipulation of the semantic parameters. The parameters are processed with large-language-model-based refinement to finalize the prompt for scene images generation. The 3D modeling module converts generated images into 3D geometries and render with post-effects through shader programming. By providing an integrated solution for generating and immersing in complex environments, our framework promotes deeper user engagement in virtual space creation, demonstrates potential for applications in VR, AR, and video game.