By Topic

Online Experiments: Lessons Learned

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$33 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

2 Author(s)

The Web provides an unprecedented opportunity to evaluate proposed changes or new features quickly using controlled experiments. The simplest experiments randomly assign live users to one of two variants: the control, which is commonly the existing version, and the treatment, which is usually a new version being evaluated. The overall evaluation criterion can be a simple metric that summarizes important business goals or a weighted combination of metrics, as is often used in credit scores. Randomization is too important to be left to chance. A common way to maintain user experience consistency is to employ a hashing function on a user ID stored in a cookie. Cryptographic hashes such as MD5 are generally the best. Failure to randomize properly can confound results when running multiple tests simultaneously. The other common mistakes in online experiments are launching a feature that is statistically significantly different but has little business value and launching a feature because it doesn't negatively impact users. Online experiments, whether they fail or succeed, generate insights that can bring a quick return on investment and promote innovation.

Published in:

Computer  (Volume:40 ,  Issue: 9 )