Impact of Quality Aspects on Overall Quality for Dialog Systems in Different Categories

Models for prediction of user judgments are useful in semiautomatic evaluation of dialog systems. So far, little attention has been paid to which system characteristics determine the weighting of individual quality aspects in the overall judgment. The purpose of our study is to fill this gap by analyzing a database covering many different systems. More specifically, 43 dialog systems are profiled based on various system features. Conducting cluster analysis, our findings suggest 6 specific categories of dialog systems. For each detected group, fourteen quality aspects collected on a questionnaire are ranked, and the most relevant aspects are selected to fit a linear regression model estimating the judgment on overall impression. The performance for each estimation model is evaluated and compared to the performance of a baseline model. Our findings show that the quality aspects selected for each estimation model differ between the cluster groups, but do not always reflect the system characteristics in that group. The model performance exceeds the performance of the baseline model in two clusters.