Towards a Set of Heuristics for Evaluating Chatbots

Chatbots are artificial intelligence tools that interact with people in different contexts. A chatbot can be useful to streamline daily processes, serve customers 24 hours a day, provide information about classes, among other things. The appearance of new development technologies has made creating a chatbot an increasingly fast and straightforward process, bringing this kind of applications to people who had never considered using them before. However, this speed in development can lead to specific problems, many of them caused by the lack of usability evaluations. Heuristic usability evaluations are user interface review processes carried out by experts and are an essential part of any assessment process. To date, there are no heuristics to evaluate the usability of chatbots. Therefore, this work proposes five usability heuristics in chatbots that come from the experience developing this type of applications, as well as from a broad review of state of the art. The set of heuristics was tested using a case study with the help of five experts, who evaluated an education-oriented chatbot. The results revealed that, although the proposed heuristics need refinement, they are an excellent first step in broadening the horizon of usability evaluations in chatbots.