A Survey of Dialogue System Evaluation

Dialogue systems provide a very efficient way for humans to interact with computer systems to access various resources and services. The evaluation of a dialogue system can provide valuable feedback for improving the system, but it is a challenging task, so attracts lots of attention from researchers. In this paper, we survey some essential criteria and widely used methods for evaluating dialogue systems, focusing on the latest research progress on this topic. Notably, we discuss machine learning based evaluation method and deep learning based ones. We also compare their advantages and disadvantages. Besides, by analysing the difficulties and challenges that the researchers face in evaluating dialogue systems, we prospect the development tendency of the research on evaluating dialogue systems.