Bayesian A/B Testing with Covariates

Randomized controlled trials (RCTs), such as A/B testing, are the gold standard for causal discovery. The statistical inference of RCTs focuses on two questions: whether the intervention has an actual effect on the target variable, and how much the impact is. We propose a novel strategy to incorporate covariates as higher-dimensional information in the Bayesian framework to improve the accuracy of statistical inference. It is guaranteed that the estimation risk would not increase with the incremental number of covariates. A simulation study shows that the mean squared error of treatment effect estimation decreases significantly after incorporating one covariate. By real-life experiment data, the estimation error of average effect size with 5 covariates decreased by 33.3% compared to not using covariate, and decreased by 98.0% compared to only using the observed value. The strategy proposed in this work is of flexibility and has the potential to be integrated with other recent studies such as experiment splitting, which may lead to further advancements in statistical inference using the Bayesian framework.