Locally Adapted Immersive Environments for Distributed Music Performances in Mixed Reality

The integration of network-based music performance with immersive media technology opens new compelling avenues for live collaborative multimedia concerts and exhibitions. This paper presents a development workflow for creating cohesive immersive environments for multimedia network music performances, where an exhibition node is connected to a remote ensemble. These environments rely on the usage of local characterization data, such as room acoustic measurements and digital twin assets, to render a mix of audio and motion-capture streams, capable of merging remote and local performers into a single shared cohesive display streamed to eXtended Reality devices. Spatialization and auralization techniques are used to add realism to the auditory elements and ground them in the acoustics of the local exhibition node shared by musicians on stage and audiences. The resulting “hybrid” display combines virtual-reality and mixed-reality principles to create a specific type of novel interactive concert experience.