Quantifying the reliability of software: statistical testing based on a usage model

When a population is too large for study, as is the case for all possible uses of a software system, a statistically correct sample must be drawn as a basis for inferences about the population. In statistical testing of software based on a Markov chain usage model, the rich body of analytical results available for Markov chains provides numerous insights that can be used in test planning. Further, the connection between Markov chains and operations research techniques permits a Markov usage model to be expressed as a system of constraints, with mathematical programming used to generate the optimal model for a particular objective function. Since a software usage model is based on the specification, all analyses may be performed early in the development cycle and used as a quantitative basis for management decisions. These techniques have been reduced to engineering practice and used in large projects by IBM, Ericsson, all branches of the US military, and others. In this paper, statistical experiments, Markov models, and optimization techniques are shown to provide a sound theoretical and practical basis far quantifying the reliability of software.