Model validation using simulated data

Effective and accurate reliability modeling requires the collection of comprehensive, homogeneous, and consistent data sets. Failure data required for software reliability modeling is difficult to collect, and even the available data tends to be noisy, distorted and unpredictable. Also, the complexity of the real world data might obscure the properties of the reliability models which are based on simpler assumptions. These properties may be revealed by evaluating the models using simpler data sets. Towards this end, we have created 20 sequences of interfailure times each from five software reliability models using rate-based simulation technique, and validated the models using the simulated data sets. In this paper we describe the experimental setup, model validation results, and the lessons learned during the experiment. Having established the credibility of simulation to generate failure data, we also show how the failure process underlying a failure data set can be described more accurately by simulating it using a combination of reliability models, as opposed to a single model as per conventional analytical techniques.