Real Haptics: Using Physical Manipulation to Control Virtual Worlds

Typical applications in mixed reality use a simulated mechanism for manipulating objects in the virtual environment, through VR controllers held in user —char63s hands, without real touch. As an alternative, within our Distributed Reality concept [3] we propose Real Haptics, a novel interaction method based on two key ideas: providing real user embodiment in the virtual spaces by segmenting user real hands and allowing the real manipulation of physical objects, which in the virtual scene may preserve or change their real appearance. To evaluate the concept, we deployed a functional prototype with Unity, using color-based algorithms for object segmentation [2] and Aruco library for object tracking [1]. Based on this platform we built a gamified testbed: a five-minute escape room game, which 53 users played twice, once with Real Haptics and once with purely virtual objects and avatars using VR controllers. After each run, users answered a survey derived from standard questionnaires measuring presence, embodiment and quality of experience. This information plus some quantitative data related to performance, validated our hypothesis that Real Haptics significantly improves presence or embodiment with respect to the counterpart virtual solution. We foresee our technology to provide a significant advance in the control of virtual environments, with a first potential focus on training applications.