From test count to code coverage using the Lognormal Failure Rate

When testing software, both effort and delay costs are related to the number of tests developed and executed. However the benefits of testing are related to coverage achieved and defects discovered. We establish a novel relationship between test costs and the benefits, specifically between the quantity of testing and test coverage, based on the Lognormal Failure Rate model. We perform a detailed study of how code coverage increases as a function of number of test cases executed against the 29 files of an application termed SHARPE. Distinct, known coverage patterns are available for 735 tests against this application. By simulating execution of those tests in randomized sequences we determined the average empirical coverage as a function of number of test cases executed for SHARPE as a whole and for each of its individual files. Prior research suggests the branching nature of software causes code coverage to grow as the Laplace transform of the lognormal. We use the empirical SHARPE coverage data to validate the lognormal hypothesis and the derivation of the coverage growth model. The SHARPE data provides initial insights into how the parameters of the lognormal are related to program file characteristics, how the properties of the files combine into properties of the whole, and how data quantity and quality affects parameter estimation.