Loading [a11y]/accessibility-menu.js
A Unified Framework for Detecting Domain and Intent Misclassifications in Large-Scale Dialogue Systems | IEEE Conference Publication | IEEE Xplore

A Unified Framework for Detecting Domain and Intent Misclassifications in Large-Scale Dialogue Systems


Abstract:

The popularity of large-scale dialogue systems such as Google Assistant, Siri, and Bixby has grown considerably in recent years. However, these systems often make errors ...Show More

Abstract:

The popularity of large-scale dialogue systems such as Google Assistant, Siri, and Bixby has grown considerably in recent years. However, these systems often make errors in domain and intent classification, resulting in incorrect responses and a suboptimal user experience. In this paper, we propose a solution that utilizes unsupervised data augmentation for consistency training (UDACT) to analyze virtual assistant’s user logs and identify instances where the system has misclassified the user’s intended domain or intent. We also analyze the characteristics of various domains, considering factors such as noise, simplicity, diversity, complexity, density and sparsity, and apply augmentation techniques accordingly to improve the system’s performance. Our solution also generates a report that highlights instances of misclassification and provides insights for the types of errors that the system is making. We conduct empirical analysis by comparing the suitability & performance with various state-of-the-art methods. Currently, our solution has achieved an accuracy of 92.4% on our In-House dataset that comprises of 30 domains and more than 300 intents and an accuracy of 98.2% on standard SNIPS-MultiWOZ2.2 dataset that comprises of 14 domains and 37 intents. This paper demonstrates the effectiveness of our solution on virtual assistant’s multi-domain user logs, showing its ability to identify misclassifications in large scale dialogue systems.
Date of Conference: 21-21 September 2023
Date Added to IEEE Xplore: 16 October 2023
ISBN Information:
Conference Location: Bangalore, India

Contact IEEE to Subscribe

References

References is not available for this document.