A Bayesian ranking and selection problem with pairwise comparisons

We consider a ranking and selection problem where sampling of two alternatives at once is required for learning about the true performances of the individual alternatives. The true performance of an alternative is defined as its average probability of outperforming the other alternatives. We derive and numerically compare four different solution approaches. Two Knowledge Gradient sampling policies are compared with a pure exploration policy and with a knockout tournament. The knockout tournament serves as a natural benchmarking approach with respect to pairwise comparisons, and determines the sampling budget provided to the other approaches. Our numerical results show that the Knowledge Gradient policies outperform both knockout tournament and pure exploration, and that they lead to significant improvements already at a very small number of pairwise comparisons. In particular we find that a nonstationary Knowledge Gradient policy is the best of the considered approaches for ranking and selection with pairwise comparisons.