On the Experimental Process in Evaluations of Brazilian Conversational Agents in Education

Conversational agents represent applications that interact in a natural language with humans. Due to their ability to process the human language, they have been explored in diverse areas, e.g., e-commerce, health, entertainment, and education. Particularly in the education field, they have been used as mechanisms that support the teaching of a second language, that recommend educational resources, and that resolve students’ doubts, as a learning partner. Brazil is particularly interested in the educational technology community studying such software systems. Despite the strong interests in their development, few studies have addressed the impact such systems exert when they are used by students. Experimental studies have enabled a better understanding of the impact of technology on education based on evidence. This article presents an overview of empirical research on conversational agents in education through a mapping of the Brazilian community, identifying the way researchers systematically evaluate such agents produced in their research work. The results show a lack of experimental methodologies and taxonomies that support the planning, development, and reporting of experimental studies. However, a set of variables commonly employed has been recognized. An initial set of guidelines has been developed towards guiding experiments in this context and supporting the systematization of studies on experimental evaluations of pedagogical conversational agents.