Leveraging GenAI for an Intelligent Tutoring System for R: A Quantitative Evaluation of Large Language Models

The tremendous advances in Artificial Intelligence (AI) open new opportunities for education, with Intelligent Tutoring Systems (ITS) powered by Generative Artificial Intelligence (GenAI) proving to be a promising prospect. Because of this, our work explores state-of-the-art (SOTA) ITS approaches with the integration of Large Language Models (LLMs) to improve programming education. We investigate whether and how a GenAI-based ITS can effectively support students in learning R programming skills. We measured the performance of three current pairings of LLMs and user interfaces: GPT-3.5 via ChatGPT, PaLM 2 via Google Bard, and GPT-4 via Bing. Therefore, we evaluated the LLMs on four types of problem settings when learning/teaching programming. Our experimental results show that the use of generative AI, specifically LLMs for R programming, is promising, where GPT-3.5 yielded the most satisfactory results. Furthermore, the advantages and limitations of our approach are addressed and revealed. Finally, open research directions towards explainable AI (XAI) and integrated self-assessment are pointed out.