Continuous Monitoring of Data in Online Randomized Experiments

Conversion rate optimization can be effective in improving the conversion rate of e-commerce site, desktop, or mobile application. This process involves creating new hypotheses about updating elements of a website or application that can be perfected, and then accepting or rejecting those hypotheses through online randomized experiments with two or more groups of users. In the classical statistical tests, the number of users is specified in advance. In the extension of Wald’s sequential analysis, the number of users is not predefined and is determined by analyzing statistical data obtained sequentially. The main contribution of this paper is a suggestion to leverage a data-driven Bayesian approach to estimate parameters related to the normal probability density function to reduce variance in closed-form solution and deliver fast business decision. Sequential analysis is especially useful for testing hypotheses about changes to the user interface, content management systems and customer support systems in online software domain.