Validation of trace-driven simulation models: more on bootstrap tests

Trace-driven or correlated inspection simulation means that the simulated and the real systems have some common inputs (say, arrival times) so the two systems' outputs are cross-correlated. To validate such simulation models, this paper formulates six validation statistics, which are inspired by practice and statistical analysis; for example, the simplest statistic is the difference between the average simulated and real responses. To evaluate these validation statistics, the paper develops novel types of bootstrapping based on subruns. Three basic bootstrap procedures are devised, depending on the number of simulation replicates: one, two, or more replicates. Moreover, for the case of more than two replicates the paper considers conditional versus unconditional resampling. These six validation statistics and four bootstrap procedures are evaluated in extensive Monte Carlo experiments with single-server queueing systems. The main conclusion is that bootstrapping of the simplest validation statistic gives the correct type I error probability, and has relatively high power.