As Secure as Dangerous Can Be: Considerations for Secure Auto-Graders in the Context of MOOCs

In the context of programming education, so-called auto-graders allow learners to receive automated feedback on their submissions. Because assessing learners' code typically involves executing the learners' untrusted code, this commonly used mechanism poses a significant security risk for these systems. Since auto-graders are mostly employed in the context of large-scale learning environments, such as universities or Massive Open Online Courses (MOOCs), security considerations are especially important. In this paper, we first introduce our auto-grader CodeOcean, which is regularly used in MOOCs with thousands of active learners, and in university contexts. As the execution of untrusted code can entail severe security implications, ensuring that the application contains no security vulnerabilities is essential. Hence, we partnered with a security consultancy to assess our auto-grader system landscape through a professional penetration test. This work presents the findings and countermeasures resulting from the performed security analysis for CodeOcean. We contextualize overarching enhancements for three main categories of threat vectors to auto-grader systems. Implementing these in any auto-grader system can improve the security and prevent learners from manipulating the assessment of their code. We also discuss the potential consequences of hardening an auto-grader, such as a reduced system performance. Therewith, we provide valuable recommendations for educators, researchers, and system designers to improve the security of auto-graders in the future, supporting their usage in even larger settings or in the context of exams.