Maximal expectation as upper confidence bound for multi-armed bandit problems

State of the art algorithms for stochastic multi-armed bandit problem are based on either one of the two principles: optimism in face of uncertainty, and probability matching. In this paper, we provide a unified approach to combine these principles. The major result is a new upper confidence bound formula, UCB-max, by which the p-value of an arm's UCB value roughly matches the arm's probability of been optimal. Our numerical study comparing UCB-max with other competitors (UCB-tuned, UCB-V, MOSS, KL-UCB and Thompson sampling) shows that UCB-max is remarkably efficient and stable.