Multisensory musical entertainment systems

As a result of our research into multisensory perception, we developed two unique musical entertainment systems - Bubble Bumble and Magic Music Desk (MMD) - that provide users with multisensory experiences in creating and playing back music in 3D physical environments through augmented reality interfaces. During the process of creating and playing back music, users encounter visual-speech, visual-tactile, and tactile-auditory experiences. We applied visual, auditory, and tactile sensory modalities to create a system that could express a wide range of human-musical artifacts. During the process of creating and playing back music, users experience a unique sensing experience of 3D-visual-speech, 3D-visual-sound - music and nonspeech sound - and tactile-auditory perceptions. Users also can experience visual-tactile sensation by using their hands to cause a corresponding system action. Beyond the multi-sensory experience, our multimedia systems also emphasize tangible human-computer interactions and communications within a physical environment. In this article, rather than focus on the application's design, we examine the systems' multi-modal perceptions. We describe each system and how it works, and we present the results of a formal user study we conducted.