Collaborating Smartwatches and HMDs to Enhance User Perception for Touch Manipulation in XR

In the realm of virtual object manipulation, multi-touch solutions have been explored since the early days of tabletop gesture interactions. However, with the advent of virtual and mixed reality, the exploration of touch-based solutions has predominantly shifted towards gesture design and input mediums like wearables, overlooking optimization for user experiences in immersive XR environments.In this paper, we introduce a novel approach by collaborating a smartwatch with AR glasses, considering both input device posture and the user’s viewpoint, to implement five levels of exploration solutions. Through user experiments, we compare and identify the two most prominent ones and summarize factors influencing users’ perception in manipulating virtual objects. These will promote more targeted interaction design and experimental research.