Crowdsourcing for Spoken Dialog System Evaluation

In this chapter, the authors describe their work in developing a crowdsourcing methodology for spoken dialog system (SDS) evaluation through Amazon Mechanical Turk (MTurk), a popular crowdsourcing marketplace that makes use of human intelligence online to perform tasks which cannot be completed entirely by computer programs. The chapter is organized as follows. The first section discusses prior work in crowdsourcing for dialog and speech assessment. The second section provides an overview of related work on SDS evaluation. The third section presents information about the experimental corpus and how automatic dialog classification is done. The fourth section presents the methodology in the use of crowdsourcing for collecting user judgments on SDSs. The fifth section presents an analysis of collected results, which demonstrates that the crowdsourcing method for the collection of user judgments is efficient, flexible and cost‐effective.