Beyond the Like Button: Deriving User Feedback with Rubrics and Metrics in Virtual Assistants

Multi-domain virtual assistants are integral to user’s daily lives, as they help with a variety of tasks and interactions. Because of their significant role, it is important to continuously monitor their performance to make sure they accurately understand what users need and can effectively execute tasks. While user feedback is critical for assessing their satisfaction levels, relying solely on explicit user feedback is impractical. To address this challenge, we propose a dual-phase framework, called a System for Performance Evaluation with Chat and Task-oriented RUbrics and Metrics (SPECTRUM), designed to extract implicit user feedback from interactions between users and virtual assistants. SPECTRUM assesses both chat-and-task-oriented domains, streamlining the evaluation process and reducing the need for disparate algorithms. In our framework, a predefined set of rubrics and metrics is utilized to assess the user-virtual assistant conversations (Phase 1), where the rubric/metric results serve as decisive elements in determining whether a conversation yields positive or negative user feedback (Phase 2), positive user feedback refers to user satisfaction and negative user feedback refers to user dissatisfaction for the conversation. To obtain results for majority of rubrics and metrics, we use the capabilities of GPT4 in zero-shot and few-shot settings with refined prompts acquired by the prompt ensembling approach. For other metrics where GPT4 gives sub-optimal results, traditional open-source models are employed. Through analysis of past conversations between users and virtual assistants, SPECTRUM extracts reasons for negative user feedback, aiding developers in identifying areas for improvement and optimizing underlying models supporting virtual assistants. Our approach achieves 80.8% accuracy on our In-House dataset and 76.9% accuracy on the SNIPS-MultiWOZ2.2 dataset, surpassing complex, costlier baseline methods. With SPECTRUM, our goal is to introduce a straightforward yet effective approach for evaluating multi-domain virtual assistants, aiming to enhance their performance and increase user satisfaction.