Bootstrap ranking & selection revisited

Many ranking-and-selection (R&S) procedures have been invented for choosing the best simulated system; in this paper we consider indifference-zone procedures that attempt to provide a probability of correct selection (PCS) guarantee. To obtain the PCS guarantee, existing procedures nearly always exploit knowledge about the particular combination of system performance measure (e.g., mean, probability, quantile) and assumed output distribution (e.g., normal, exponential, Poisson). In this paper we take a step toward general-purpose R&S procedures that work for many types of performance measures and output distributions, including situations in which different simulated alternatives have entirely different output distributions. There are only two versions of our procedure: with and without the use of common random numbers, and they can be applied to performance measures that can be expressed as expected values or quantiles. To obtain the desired PCS we exploit intense computation via bootstrapping, and establish the asymptotic PCS under very mild conditions. We also report results of an empirical study to assess the procedures' small-sample properties.