Automating Pipelines of A/B Tests with Population Split Using Self-Adaptation and Machine Learning

A/B testing is a common approach used in industry to facilitate innovation through the introduction of new features or the modification of existing software. Traditionally, A/B tests are administrated manually and conducted sequentially, with each experiment targeting the entire population that uses the corresponding application. This approach can be time-consuming and costly, particularly when the experiments are not relevant to the entire population. To tackle these problems, we present a new self-adaptive approach called AutoPABS, short for Automated Pipelines of A/B tests using Self-adaptation. AutoPABS contributes: (1) a self-adaptive architecture that enables automated execution of A/B testing pipelines, (2) a population split component that is managed by the managing system of the self-adaptive architecture to enable more efficient A/B testing pipeline execution, and (3) a notation to specify A/B testing pipelines that can be executed using the self-adaptive architecture. We started the evaluation with a small survey to probe the appraisal of the notation and infrastructure of AutoPABS. Then we performed a series of tests to measure the gains obtained by applying a population split in an automated A/B testing pipeline, using an extension of the SEAByTE artifact. The survey results show that the participants appraise the usefulness of automating A/B testing pipelines and population split. The test results show that automatically executing pipelines of A/B tests using the self-adaptation architecture with population split accelerates the identification of statistically significant results by exploiting parallel A/B test execution compared to traditional sequential execution of A/B tests.