Extra-normal interactions in mediated virtual environments: An investigation of an audio-visual crossed-sense modality

Here, we propose that the use of unusual crossed-sense-modality couplings can be exploited to engineer non-realistic but potentially beneficial VEs providing high degrees of interaction and presence, and provide new avenues for enhancing the experience and utility of single- and multi-user VE technologies. We report user's reactions to mediated environments in which the sound of their voice is represented via a frequency-to-color-mapped visualization. Users reported a sense of interaction and enjoyment, but results from a preliminary user study have yet to show a clear link with presence. We discuss the next stages of the research which will investigate how mixed-sense modalities contribute to co-presence and collaboration in multi-user environments.