Supporting Virtual Collaboration in Spatial Design Tasks: Are Surrogate or Natural Gestures More Effective?

Hand gestures offer a rich source of information for communication of spatial concepts. However, gestures are often lost when people work together in virtual space. Gestures may be conveyed in virtual spaces through many approaches, for example by drawing circles and arrows around objects on a shared screen, or using natural, real-time videos of a distant collaborator's hands or body. The first is an example of “surrogate” gestures and the second of “natural” gestures. The first is relatively inexpensive but requires users take some effort to draw and erase temporary symbols; the second is more costly but also more expressive and natural. This work will: 1) provide a brief, high-level framework characterizing tools that support virtual gestures; and 2) present an experimental study comparing the cognitive effort required when using sketched surrogate gestures versus natural gestures in a virtual mechanical design tasks. The larger goal is to understand whether the additional cost to support natural gestures is warranted, thus informing the design of future virtual collaboration tools. We recruited 24 participants and asked them to collaborate using three different approaches: face-to-face (as a baseline), virtual-sketching (which allows users to make gestural sketches on their drawings), and virtual-sketching with natural hand videos projected on the drawing surface. We found that users' cognitive workload was significantly reduced when natural hand videos were added to a virtual-sketching environment. These results suggest that the ease and expressivity of natural gestures provide more benefits than simpler surrogate gestures for virtual mechanical design tasks.