Creating an Olfactory Mask Dataset using Computer Vision and LLMs for Virtual Reality Environments

Integrating scent into Virtual Reality (VR) enriches immersion by linking olfactory elements with visual cues, but requires sophisticated data modeling and segmentation techniques. To address these challenges, we propose the concept of 'olfactory masks', which involves creating image segmentation associated with specific scents, thereby enhancing sensory experiences in VR. This study advances VR immersion by linking olfactory and visual elements using multimodal AI models and sophisticated segmentation. Using CLIP with 115 olfactory labels from the OlfactionBase dataset, prompts were framed to enhance scent-focused recognition, though CLIP's limitations highlighted the need for olfactory-specific training. 360° panoramic scenes were generated using ADE20K scene descriptions refined by GPT-4o mini, and processed with Skybox AI to create diverse immersive visuals. By segmenting images with scent labels, specific odors were mapped to regions within the virtual environment. The results show that the integration of language, vision and olfactory models significantly enhances VR, supporting deeper sensory experiences.