Estimation of Skill Distributions

In this paper, we study the problem of learning the skill distribution of a population of agents from observations of pairwise games in a tournament. These games are played among
n
randomly drawn agents from the population. The agents in our model can be individuals, sports teams, or even Wall Street fund managers. Formally, we postulate that the likelihoods of outcomes of games are governed by the parametric Bradley-Terry-Luce (or multinomial logit) model, where the probability of an agent beating another is the ratio between its skill level and the pairwise sum of skill levels, and the skill parameters are drawn from an unknown, non-parametric skill density of interest. The above problem is, in essence, to learn a distribution from noisy and quantized observations. We propose a surprisingly simple and tractable algorithm that learns the skill density with near-optimal minimax mean squared error scaling as
n
^-1+ε
, for any ε > 0, so long as the density is smooth. Our approach brings together prior work on learning skill parameters from pairwise comparisons with kernel density estimation from non-parametric statistics. We then prove information theoretic lower bounds which establish minimax near-optimality of the skill parameter estimation technique used in our algorithm. These bounds utilize a continuum version of Fano’s method along with a careful covering argument. Furthermore, we show that estimation error bounds for the skill density translate to theoretical guarantees on estimating the differential entropy and other bounded statistics of the skill density. Finally, we apply our algorithm to data from soccer world cups and leagues, cricket world cups, and even mutual funds. We find that the differential entropy of a learnt distribution provides a quantitative measure of overall skill in a tournament, which in turn can provide explanations for popular beliefs about perceived qualities of sporting and other tournaments.