A Quality of Experience Evaluation Comparing Augmented Reality and Paper Based Instruction for Complex Task Assistance

Augmented reality (AR) can support a user in performing an expert task by overlaying real world objects with the domain specific information required to complete the task. Understanding how users can process and use such information is very important for informing the design of AR technologies and applications. In this paper, the results of a quality of experience (QoE) evaluation of an AR application for the task of solving a Rubik's Cube are presented. The Rubik's Cube was selected based on its familiarity and the expertise needed to solve it unaided. An empirical approach was taken to identify the QoE features that affect the usability and utility of an AR head-mounted display (HMD) compared with paper-based instruction. The QoE evaluation methodology involved the capture and analysis of implicit and explicit QoE metrics. The utility (in terms of performance) of each mode of instruction was objectively measured using: (a) cube completion success rates; and (b) time-to-completion. The implicit metrics of electrodermal activity (EDA), skin temperature, heart rate and the novel use of facial action units (AUs) were recorded to infer emotional state during the task completion. Finally, with respect to explicit metrics, the test subjects completed a Likert scale questionnaire post the experience to subjectively report QoE as well as a self-assessment manikin (SAM) questionnaire to self-report emotional state upon task completion. The results show that AR yielded higher success rates and significantly lower time-to-completion rates. The AR group explicitly reported higher levels of positive valance (affective state) than the paper-based group. The physiological data showed that the AR group were less stressed (via EDA) than the paper-based group. Finally, analysis of the AU data reflected a greater than chance (total: 21.85%) accuracy when predicting affective state based on SAM questionnaires as ground-truth.