Online Experiments: Lessons Learned

The Web provides an unprecedented opportunity to evaluate proposed changes or new features quickly using controlled experiments. The simplest experiments randomly assign live users to one of two variants: the control, which is commonly the existing version, and the treatment, which is usually a new version being evaluated. The overall evaluation criterion can be a simple metric that summarizes important business goals or a weighted combination of metrics, as is often used in credit scores. Randomization is too important to be left to chance. A common way to maintain user experience consistency is to employ a hashing function on a user ID stored in a cookie. Cryptographic hashes such as MD5 are generally the best. Failure to randomize properly can confound results when running multiple tests simultaneously. The other common mistakes in online experiments are launching a feature that is statistically significantly different but has little business value and launching a feature because it doesn't negatively impact users. Online experiments, whether they fail or succeed, generate insights that can bring a quick return on investment and promote innovation.