Difference Grouping and Test Suite Evaluation: Lessons from Automated Differential Testing for Adobe Analytics

At Adobe, we have learned that an important measure of software quality is the ability to acquire and retain customers. We have found that, in Adobe Analytics, a good indicator of that ability is the number of customer-reported regression bugs. To keep this number low we have found that the best testing approach is differential testing (DT) [4] . Hundreds of teams at Google are using DT, and describe it as the only ‘sane’ way to do large scale complex verification [5] . One of the greatest challenges for DT is how to handle the large number of differences that are often discovered. DT can only tell you if behavior has changed but cannot tell you if those changes are acceptable. Traditionally, a large manual effort is required to inspect each difference to determine the quality of the code change(s). One approach used at Adobe to solve this problem is to create difference groups. Grouping accomplishes two important tasks: (1) A large reduction in the number of differences that must be examined manually; and (2) A simple and practical method of test suite evaluation. By grouping differences, only one test case from each group needs be evaluated for correctness. Further, analyzing groups reveals how effective the test suite has been in exercising code changes. If differences are expected and we fail to find groups that have these differences, then we know that our test suite is incomplete and needs attention.