Using Randomized Controlled Trials in eLearning: How to Add Content A/B Tests to a MOOC Environment

RCTs are an essential part of scientific research, allowing testing of different experimental setups with independent groups. In an eLearning context, this can relate to both platform features and teaching didactic experiments. So far, the MOOC platform we operate provided no simple way to run A/B tests at the course content level, but only at the platform feature level. However, this limitation is unacceptable when studying the effect of different content presentation styles and methods, so we had to develop a new RCT feature. In this paper, we describe our requirements and the challenges associated with content A/B testing in MOOCs. We examine how other platform providers solved these problems and compare our solution design. We then describe the technical implementation of the new feature, which difficulties we encountered on our complex infrastructure, and how we solved them. We also share our results from the first successful content A/B test experiments and further planned enhancements. The paper thus offers researchers the opportunity to better understand the technical concept and our implementation of content A/B testing on a MOOC platform. It helps developing a technical solution for RTCs on their infrastructure.