Human Evaluation of GPT for Scalable Python Programming Exercise Generation

Online coding platforms (OCPs) often offer a limited selection of exercises, which can restrict the scope of Computer Science (CS) education. This study investigates the capabilities of Large Language Models (LLMs), particularly GPT-4 Turbo, in broadening this scope by autonomously generating Python programming exercises. These exercises are tailored to the CS1 curriculum-an introductory course in computer science. Utilizing curriculum-driven prompt engineering, we developed a dataset of 11,700 exercises, characterized by a variety of cate-gories, types, and difficulty levels. These exercises are distributed across 78 unique topics, which were derived from the CS1 course catalogs of leading universities and supplemented with online educational resources. To evaluate the effectiveness of GPT-4 Turbo in generating CSI Python programming exercises, we conducted a user study involving both students and instruc-tors. The study focused on several metrics: exercise quality, curriculum relevance, understandability, appropriate difficulty level, and the generation of useful hints. Our findings indicate that GPT-4 Turbo can produce high-quality, educationally effective programming exercises at scale, provided that the prompts are systematically crafted. Based on insights from the user study, adjustments to prompt design are recommended to optimize exercise generation. Our research concludes that GPT-4 Turbo can be seamlessly integrated into AI-driven OCPs, offering a scalable, cost and time-effective method to enhance CS edu-cation. This is achieved through targeted prompt engineering and thorough data preprocessing to mitigate inconsistencies. The code is available online: https://github.com/DSAatUSU/GPT_CS1400_Exercise_Generation