Predicting user evaluations of spoken dialog systems using semi-supervised learning

User evaluations of dialogs from a spoken dialog system (SDS) can be directly used to gauge the system's performance. However, it is costly to obtain manual evaluations of a large corpus of dialogs. Semi-supervised learning (SSL) provides a possible solution. This process learns from a small amount of manually labeled data, together with a large amount of unlabeled data, and can later be used to perform automatic labeling. We conduct comparative experiments among SSL approaches, classical regression and supervised learning in evaluation of dialogs from CMU's Let's Go Bus Information System. Two typical SSL methods, namely co-training and semi-supervised support vector machine (S3VM), are found to outperform the other approaches in automatically predicting user evaluations of unseen dialogs in the case of low training rate.