Multimodal Evaluation of the Differences between Real and Virtual Assemblies

What are the technological bottlenecks in virtual assembly simulations with haptic feedback? To tackle this question, we present an evaluation study in which real feedback modalities are gradually replaced by synthetic ones. In particular, the effects of the following factors on the user performance and perception during virtual assemblies are analyzed: (i) a visual feedback system consisting of an nVisor head-mounted display, (ii) our haptic device HUG suited for unscaled upper-body movements, and (iii) our novel six-DoF constraint-based haptic rendering algorithm. Besides that, the influence of (iv) real collision sounds is also examined to a lesser extent. The experimental assembly scenario consisted of three variations of peg-in-hole tasks which were performed by a total of N = 24 participants in a within-design study. The mentioned three synthetic factors (i)-(iii) gradually replaced in five degrees or steps the real feedback sources, ending up in completely virtual assembly simulations. For each of the degrees, three objective variables (completion time, collision forces, and muscular effort) and five subjective ratings (related to the perception of realism and the workload) were recorded and statistically analyzed. In order to explain subjective perception also with objective measures, reaction times of a secondary audio task performed in parallel with the assembly exercises were recorded, too. While previous works have mainly focused on differences of completion time between real and virtual manipulations, our results show how all of the mentioned twelve performance and perception indicators are influenced by each of the four varied feedback factors, building a multi-modality relationship function that maps our or similar systems and expected user responses. In general, the haptic feedback modality turned out to have the largest impact on the dependent variables, particularly the HUG interface, whereas audio cues seemed to be less significant. We quantify these previous and further qualitative statements within the domain defined by the used systems. Moreover, the relationship of our insights with related other work is discussed, and their projections are outlined.