By Topic

Multisensory musical entertainment systems

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$33 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

7 Author(s)
Zhiying Zhou ; National Univ. of Singapore, Singapore ; A. D. Cheok ; Wei Liu ; Xiangdong Chen
more authors

As a result of our research into multisensory perception, we developed two unique musical entertainment systems - Bubble Bumble and Magic Music Desk (MMD) - that provide users with multisensory experiences in creating and playing back music in 3D physical environments through augmented reality interfaces. During the process of creating and playing back music, users encounter visual-speech, visual-tactile, and tactile-auditory experiences. We applied visual, auditory, and tactile sensory modalities to create a system that could express a wide range of human-musical artifacts. During the process of creating and playing back music, users experience a unique sensing experience of 3D-visual-speech, 3D-visual-sound - music and nonspeech sound - and tactile-auditory perceptions. Users also can experience visual-tactile sensation by using their hands to cause a corresponding system action. Beyond the multi-sensory experience, our multimedia systems also emphasize tangible human-computer interactions and communications within a physical environment. In this article, rather than focus on the application's design, we examine the systems' multi-modal perceptions. We describe each system and how it works, and we present the results of a formal user study we conducted.

Published in:

IEEE MultiMedia  (Volume:11 ,  Issue: 3 )