Sonification of Surface Tapping Changes Behavior, Surface Perception, and Emotion

The audio feedback resulting from object interaction provides information about the material of the surface and about one's own motor behavior. With the current developments in interactive sonification, it's now possible to digitally change this audio feedback, making the use of interactive sonification a compelling approach to shape tactile surface interactions. Here, the authors present a prototype for a sonic interactive surface, capable of delivering surface tapping sounds in real time when triggered by users' taps on a real surface or on an imagined "virtual" surface. In this system, the delivered audio feedback can be varied so that the tapping sounds correspond to different applied strengths during tapping. The authors also propose a multidimensional measurement approach to evaluate user experiences of multimodal interactive systems. They evaluated their system by looking at the effect of the altered tapping sounds on emotional action-related responses, the users' interactions with the surface, and perceived surface hardness. Results show the influence of the sonification of tapping at all levels: emotional, behavioral, and perceptual. These results have implications on the design of interactive sonification displays and tangible auditory interfaces aiming to change perceived and subsequent motor behavior as well as perceived material properties.