Novelty, Rigidity, and Complexity: Toward Developing AI-Resistant Assessments in an Introductory Computer Science Class

Since the public launch of ChatGPT on November 20, 2022, there has been significant interest in its role in education. While some analyses focus on its potential to support student learning, a parallel line of investigation examines its potential to facilitate the false demonstration of competency on assessments. A broader underlying question is whether certain skills remain valuable to teach in the age of artificial intelligence, particularly in developing a new generation of computer programmers who continue to employ critical problem-solving skills in their work. The work presented in this paper examines the impact of large language models (LLMs), such as ChatGPT-4, on a college-level introductory computing course offered simultaneously as a massive open online course (MOOC) on the edX platform. The study focuses on the strengths and limitations of LLMs in solving coding assignments while also aiming to identify problems that are resistant to LLM solutions, so these types of specific categories could be employed by other instructors in their MOOC courses to provide a better experience for students. The paper explores GPT’s proficiency in various areas, including pseudo-code interpretation, handling multiple correct answers, and addressing complex problem statements. The goal is to create a robust framework that discourages over-reliance on AI assistance from some students while preserving the scalability of the course. This research provides insights into the dynamics of AI in education and emphasizes the need for a balanced approach between technological assistance and genuine student participation.