Tailoring Your Code Companion: Leveraging LLMs and RAG to Develop a Chatbot to Support Students in a Programming Course

Students frequently rely on chatbots powered by generative Artificial Intelligence (GenAI), such as ChatGPT, Copilot, Gemini, and Claude, to assist with a wide range of academic tasks. However, these chatbots are not specifically designed for the context of particular courses, which can lead to responses that are sometimes inaccurate or insufficiently relevant. This paper introduces a chatbot specifically designed to support first-year engineering students in a Java programming course. Developed using the Retrieval-Augmented Generation (RAG) technique, the chatbot draws on course-specific resources such as videos, quizzes, programming exercises, and other materials, while using OpenAI’s Large Language Models (LLMs) GPT-4 and GPT-3.5 for information analysis and response generation. The data collected, consisting of logs from 1,059 messages sent by students to the chatbot and 30 responses to a survey, indicate that students primarily used the chatbot to clarify concepts and explain code snippets. Moreover, most of the students reported that the responses provided by the chatbot were well suited to the Java programming course.