Comparison of Claude (Sonnet and Opus) and ChatGPT (GPT-4, GPT-4o, GPT-o1) in Analyzing Educational Image-Based Questions from Block-Based Programming Assessments

This study presented a novel application of OpenAI's GPT-o1 model for analyzing educational image-based questions derived from block-based programming assessments, contributing to the first systematic exploration of this advanced AI model in this field. We evaluated GPT-o1 alongside other state-of-the-art models, including GPT-4, GPT-4o, GPT-4o mini, Claude3.5 Sonnet, and Claude-3 Opus, to assess their accuracy in solving multiple-choice and fill-in-the-blank questions from a standardized block-based programming test. Our findings demonstrated that GPT-o1 significantly outperformed other models, achieving an accuracy of 80 %, which we attributed to its enhanced image analysis capabilities. Comparative analyses revealed GPT-01's superior performance in interpreting and responding to visual prompts, particularly in tasks requiring logical reasoning and computational thinking. Furthermore, the Pearson correlation test highlighted significant relationships between GPT-o1 and GPT-4o. Our contributions highlighted the potential applications of advanced chatbots in block-based programming education. These technologies can serve as personalized learning tutors to guide students in understanding and solving block-based programming tasks independently. Additionally, they can assist educators in automating the grading process and generating tailored programming exercises, thus improving teaching efficiency.