Using the Visuo-Haptic Illusion to Perceive and Manipulate Different Virtual Objects in Augmented Reality

The prop-based 3D virtual object manipulation method is widely used for interaction in Augmented Reality (AR) due to its convenience and flexibility. However, when the represented virtual object is different from the physical prop, the look and feel of the object are not well aligned. To address this problem, we present a dynamic finger remapping approach to creating a visuo-haptic illusion that dynamically adjusts the presented virtual hand posture to fit different sizes and shapes of virtual objects in AR. The finger movement toward a physical prop is synchronously remapped to the movement of the virtual fingers towards the corresponding virtual object. We developed a system that enables users to perceive consistent visual and tactile feedback while grasping and releasing various virtual objects represented by a physical prop. We conducted a user study to explore the effect of this visuo-haptic illusion on the perceived size of virtual objects, setting the sizes of the rendered virtual object and the physical prop as independent variables. We found that the perceived size of a virtual object varied with its rendered size in an almost linear fashion, while the physical prop size did not significantly affect the perception. We also conducted a second study to compare our system with a current prop-based method on virtual object manipulation. The results indicated that the remapped hands could effectively improve the realism and naturalness of the experience.