VO.I.C.E. FIRST: Supporting Human Assistants with Real-Time Voice Understanding

While AI and automation have made significant strides in customer support, there are still situations where human intervention via voice channels is necessary to provide the best possible customer experience. In fact, although AI and chatbots have become increasingly sophisticated, they may not always be able to handle complex or nuanced customer issues. Human agents can better understand and respond to these situations, providing tailored solutions. At the same time, solving non-trivial customer problems often requires access to knowledge bases and contextual customer information, for which AI is particularly well suited. Hence the idea of integrating human and artificial intelligence in a hybrid solution. We developed an AI system to help human assistants in the process of handling conversations. This system can be viewed as a collaborative bot (cobot). The cobot captures the audio stream of the conversation, converts it to text and analyzes it in real time. The extracted tokens are classified and sent to a reasoning system based on a knowledge graph, that provides information and action suggestions to the human assistant. Assistants are also capable of providing information to the reasoning system, utilizing their human understanding of the client's circumstances as they unfold. While designing a prototypical solution for utility services, we have faced the problem of real-time use of computationally complex procedures, including spontaneous speech understanding and knowledge-based heuristic rules. Moreover, we adopted a standards-based approach and experimented with open source reasoners and publicly available language models. The paper outlines the system architecture and design, and discusses the results of the first experiments.