Using Branch Frequency Spectra to Evaluate Operational Coverage

Coverage metrics try to quantify how well a software artifact is tested. High coverage numbers instill confidence in the software and might even be necessary to obtain certification. Unfortunately, achieving high coverage numbers does not imply high quality of the test suite. One shortcoming is that coverage metrics do not measure how well test suites cover systems in production. We look at coverage from an operational perspective. We evaluate test suite quality by comparing runs executed during testing with runs executed in production. Branch frequency spectra are employed to capture the behavior during runtime. Differences in the branch frequency spectra between field executions and testing runs indicate test suite deficiencies. This post-release test suite quality assurance mechanism can be used to (1) build confidence by pooling coverage information from many execution sites and (2) guide test suite augmentation in order to prepare the test suite for the next release cycle.