Error resolution during multimodal human-computer interaction

Recent research indicates clear performance advantages and a strong user preference for interacting multimodally with computers. However, in the problematic area of error resolution, possible advantages of multimodal interface design remain poorly understood. In the present research, a semi automatic simulation method with a novel error generation capability was used to collect within subject data before and after recognition errors, and at different spiral depths in terms of number of repetitions required to resolve an error. Results indicated that users adopt a strategy of switching input modalities and lexical expressions when resolving errors, strategies that they use in a linguistically contrastive manner to distinguish a repetition from original failed input. Implications of these findings are discussed for the development of user centered predictive models of linguistic adaptation during human computer error resolution, and for the development of improved error handling in advanced recognition based interfaces.