Generative AI: How Well Can it Understand Conversational UX?

Conversational agents (CA) are growing in popularity and as such they must provide a good end-user experience (UX). Using subjective scales is one popular way to gather such UX, and the conversational UX scenario is not an exception. However, these scales do not have a common ground while identifying and naming the different UX dimensions. In this work we use a generative AI based approach for analyzing a sample of 23 well-established scales for measuring UX of CAs based on the semantic similarity of the items and group them together. Our results suggest some differences between the AI- generated UX dimensions and those established by current research. While generative AI is capable of capturing the common pragmatic aspects of UX, it falls short while comprehending the more nuanced aspects of conversational AI