Error-prone Voice and Graphical User Interfaces in a Mobile Application

This paper describes an experimental study involving a smartphone-based application using speech and touch screen as input modalities. Following our previous work on the effects of modality efficiency and input performance of voice user interfaces (VUI) and graphical user interfaces (GUI) on modality usage and perceived mental effort, we particularly investigate the so far not considered influence of touch screen errors. Participants had to perform several tasks with the mobile device in four error conditions of the two input modalities: (1) no GUI errors & no VUI errors (2) GUI errors & no VUI errors (3) no GUI errors & VUI errors (4) GUI errors & VUI errors. The results show significantly increasing speech usage in the presence of GUI errors. Regarding mental effort and perceived pragmatic quality, no differences emerge as long as one modality functions properly. However, if errors affect both, VUI and GUI, mental effort increases significantly and ratings on perceived pragmatic quality decrease.