Virtual multi-tools for hand and tool-based interaction with life-size virtual human agents

A common approach when simulating face-to-face interpersonal scenarios with virtual humans is to afford users only verbal interaction while providing rich verbal and non-verbal interaction from the virtual human. This is due to the difficulty in providing robust recognition of user non-verbal behavior and interpretation of these behaviors within the context of the verbal interaction between user and virtual human. To afford robust hand and tool-based non-verbal interaction with life-sized virtual humans, we propose virtual multi-tools. A single hand-held, tracked interaction device acts as a surrogate for the virtual multi-tools: the user's hand, multiple tools, and other objects. By combining six degree-of-freedom, high update rate tracking with extra degrees of freedom provided by buttons and triggers, a commodity device, the Nintendo Wii Remote, provides the kinesthetic and haptic feedback necessary to provide a high-fidelity estimation of the natural, unencumbered interaction provided by one's hands and physical hand-held tools. These qualities allow virtual multi-tools to be a less error-prone interface to social and task-oriented non-verbal interaction with a life-sized virtual human. This paper discusses the implementation of virtual multi-tools for hand and tool-based interaction with life-sized virtual humans, and provides an initial evaluation of the usability of virtual multi-tools in the medical education scenario of conducting a neurological exam of a virtual human.