Innovating Assessment with Conversational Agents: A Technology-Enhanced Approach to Formative Assessments

Conversational agents (e.g., ChatGPT) have become popular tools for simulating formal and casual human-like dialogues. Researchers have designed conversational agents to improve instruction quality and support student learning and investigated their effects on learning outcomes. In addition to their instructional use, conversational agents can also be incorporated into assessments to measure student learning. Students can answer items within a digital conversational environment by typing their answers while receiving feedback for correct responses and follow-up questions for incorrect responses. This study introduced a conversation-based assessment (CBA) with three selected-response and two constructed-response tests and evaluated its performance based on intent classification and confidence score. CBA was designed using Rasa and deployed to Google Chat to share with students in an undergraduate-level course. The results showed that CBA with both selected-response and constructed-response tests produced high performance and confidence scores for student responses. In particular, CBA with the selected-response format showed perfect accuracy between system design and implementation. In comparison, CBA with constructed-response items consistently matched student responses to the appropriate conversation paths for the most part. Overall, this study shows the potential of CBA as a technology-enhanced assessment tool.