Evaluation of spoken language understanding and dialogue systems

A spoken-language understanding and dialogue system in the domain of appointment scheduling is presented. The system is capable of understanding complex times, e.g. it correctly combines discontinuous constituents and resolves ambiguities. A distributed representation of surface structure models and an incremental semantic analysis are used to manage the complexity. An elaborate evaluation of the system based on measurements of accuracy was carried out. Our approach combines pattern recognition with linguistic aspects, forming a system of measurement consisting of word accuracy, constituent accuracy and concept accuracy components.