Sense of Convergence: Exploring the Artistic Potential of Cross-modal Sensory Transfer in Virtual Reality

The Sense of Convergence (SoC) project explores the artistic potential of cross-modal sensory transfer within a virtual reality context, leveraging insights from cognitive science to delve into the nature of genuine perception. In this paper, we propose a framework for cross-modal sensation parameters for assets and introduce our project showcase, a three-step experience sequence based on interaction design. To design the framework, we first conducted a preliminary psychophysical experiment that established the foundation for sensory alignment and examined the effective transfer of the smooth-rough sensation across visual, tactile, and auditory modalities. Subsequently, we designed a three-step VR experience that detaches the participant from the current reality, then gradually transfers two sensory modalities through convergence, and finally reintegrates them into reality. Feedback from participant interviews following our pilot showcase suggests that our approach successfully encourages questioning the absoluteness of reality and stimulates thought about sensory transfer and the relativity of perception. Through this work, we laid the groundwork for a VR artwork framework that investigates the concept of cognitive relativity via sensory substitution. We propose a novel approach that integrates cognitive sciences and virtual reality interaction, challenging the idea of an absolute reality as perceived through our senses.