How Useful TutorBot+ is for Teaching and Learning in Programming Courses: a Preliminary Study

Objective: The objective of this paper is to present preliminary work on the development of an EduChatBot tool and the measurement of the effects of its use aimed at providing effective feedback to programming course students. This bot, hereinafter referred to as tutorBot+, was constructed based on chatGPT3.5 and is tasked with assisting and providing timely positive feedback to students in computer science programming courses at UCSC. Methods/Analysis: The proposed method consists of four stages: (1) Immersion in the feedback and Large Language Models (LLMs) topic; (2) Development of tutorBot+ prototypes in both non-conversational and conversational versions; (3) Experiment design; and (4) Intervention and evaluation. The first stage involves a literature review on feedback and learning, the use of intelligent tutors in the educational context, as well as the topics of LLMs and chatGPT. The second and third stages detail the development of tutorBot+ in its two versions, and the final stage lays the foundation for a quasi-experimental study involving students in the curriculum activities of Programming Workshop and Database Workshop, focusing on learning outcomes related to the development of computational thinking skills, and facilitating the use and measurement of the tool’s effects. Findings: The preliminary results of this work are promising, as two functional prototypes of tutorBot+ have been developed for both the non-conversational and conversational versions. Additionally, there is ongoing exploration into the possibility of creating a domain-specific model based on pretrained models for programming, integrating tutorBot+ with other platforms, and designing an experiment to measure student performance, motivation, and the tool’s effectiveness.