A set of quantitative user experience metrics for multi-modal dialog systems

This paper introduces a framework for evaluating multimodal mobile speech applications. Existing evaluation speech frameworks from related disciplines have been extended to meet the needs of multi-modal mobile speech applications. This is done by adding the dimension of userspecific metrics both as time-averages as well as time-series metrics. This framework extension allows for in-depth user experience analysis over time, resulting in insights that can be used for system design improvements. The usability of the proposed framework has been evaluated with data from a multi-modal spoken mobile application.