Design of Experiments in Simulator Validation

A common problem encountered in computer system simulation is that of validating that the simulator can produce, with a reasonable degree of accuracy, the same information that can be obtained from the modeled system. This is basically a statistical problem because there are usually limitations with respect to the number of controlled tests that can be carried out, and assessment of the fidelity of the model is a function of the signal to noise ratio. That is, the magnitude of error which can be tolerated depends upon the size of the effect to be predicted. In this paper we describe, by example, how techniques of statistical design and analysis of experiments have been used to validate the modeling of the dispatching algorithm of a time sharing system. The examples are based on a detailed, trace-driven simulator of CP-67. They show that identical factorial experiments involving parameters of this algorithm, when carried out on both the simulator and on the actual system, produced statistically comparable effects.