On the Feasibility of Serverless Functions in the Context of Auto-Graders

Learners interested in acquiring fundamental programming skills may choose from a variety of different offers, including Massive Open Online Courses (MOOCs). Usually, these courses not only include lecture videos and multiple-choice quizzes, but also feature hands-on programming exercises, allowing learners to apply their newly acquired knowledge right away. Since solving these exercises requires access to a programming tool chain, most MOOCs embed their exercises in a web-based environment supplying necessary tools. One of these so-called auto-graders is CodeOcean, which allows learners to write and run code or receive automated feedback. While a web-based auto-grader lowers the entry barrier for learners to get started, providing sufficient resources for all code executions poses an additional challenge for the MOOC provider, especially during high-demand periods. Therefore, we evaluated serverless functions as offered by cloud computing providers for the use in auto-graders and conducted a Randomized Control Trial. Although serverless functions at first appear to be slower compared to our existing containerized execution of learners' code, they convinced with more constant execution times in high-demand periods.