Towards Automated Assessment of High School Programming

Teaching computer programming has become common in high schools, and often students' programming work needs to be assessed formally. In Aotearoa New Zealand this formal assessment happens primarily in the last three years of high school, and is currently carried out manually by teachers. Here we explore the possibility of automating this assessment with the goal of decreasing teacher workload, and increasing fairness and transparency. To do this, we first report on a multi-part survey of teachers in New Zealand schools. The first part of the survey identifies the range of languages that need to be supported. The second part of the survey, in conjunction with a literature review of automated assessment, gives us a range of tools that are designed to automatically assess programming. We evaluate this group of assessment tools based on a wide range of criteria gathered from the survey of schools and the literature review. This includes the programming languages these tools can check, with a priority given to programming languages teachers are already using in the classroom. We also consider their resilience to common programming mistakes (such as infinite while loops), whether the tools can check style and quality, whether they include gamification, whether they use closure to encourage students, what kinds of programming constraints we can put on questions, and if they are open source. Based on this analysis, CodeRunner was the contender that most strongly stood out for our context, and is the tool we will move forward with in implementing automated assessment in a New Zealand high school environment.