Do mean-based ranking and selection procedures consider systems' risk?

The legacy simulation approach in ranking and selection procedures compares systems based on a mean performance metric. The best system is most often deemed as the one with the largest (or smallest) mean performance metric. In this paper, we discuss the limitations of the mean-based selection approach. We explore other selection criterion and discuss new approaches based on stochastic dominance using an appropriate section of the distribution function of the performance metric. In this approach, the decision maker has the flexibility to determine a section of the distribution function based on the specific features of the selection problem representing either the downside risk, upside risk, or central tendency of the performance metric. We discuss two different ranking and selection procedures based on this new approach followed by a small experiment and present some open research problems.