Using Chat-GPT to Create Multiple Choice CS Exams

This work in progress Innovative Practice paper presents our work evaluating how Large Language Models (LLMs) can be utilized to aid in the development of scalable and authentic programming exam questions for students in upper-division courses. Traditionally, students' understanding of computing concepts are often verified using written exams. This is especially the case in larger universities due to their overall enrollment sizes, making more authentic, involved assignments particularly hard to facilitate. Since the rise of Chat-GPT and other LLMs, the use of such exams has seen a resurgence in popularity in classrooms globally. Fears have recently grown that programming assignments, particularly those that are take home assignments, are not adequate assessments of students' understanding given the ease at which students can use LLMs such as Chat-GPT to obtain answers. multiple-choice exams have been a traditionally popular programming exam format, where-in students are required to choose the correct answer to a prompt or the segment of code that best fits within a larger block of code. Another form of exam question that has been traditionally popular is providing students with blocks of code and ask students to either interpret the code's meaning or find the errors in the code. Both of these styles of assignments provide opportunities for students to demonstrate a deep understanding of code syntax and structure, while also tasking them to demonstrate their understanding of what the intended purpose of the code is. However, the development of these exam problems has traditionally required a substantial amount of work for instructors and teaching assistants to develop. Our goal with this work was to determine if LLMs represent effective tools for developing exam questions. Additionally, we wanted to see if LLMs could effectively design problems based on learning outcomes, allowing the instructor to become the evaluator of the exam questions rather than the original author. Our motivation for this was to develop a reproducible and easier to implement methodology for developing exams at scale for instructors while alleviating the workload imposed on instructors during the development of exams. Our initial research has indicated the use of these LLMs for exam problem generation greatly reduces the workload for instructors while allowing for the creation of far richer programming questions for students that require them to apply more of their knowledge to individual problems. We have had a good amount of success in developing these problems for upper-division courses, as well as introductory-level courses. This work represents initial steps towards the use of these LLMs for generating exams and more work is needed to determine the actual efficacy and long-term benefits and reliability of these tools. Regardless, we are confident that LLMs in their current form represent incredibly powerful tools for instructors to utilize in the development of their course exams.