Selection of learning experts

The contextual bandits can be viewed as a generalization of online classification models, where only the chosen class is observed. The selection of learning experts allows to find the best parametrization of an expert during its learning, within a set of predefined parameters, and reduces the bias of the hypothesis space, and hence improves the performances. As the contextual bandits learn, their performances tend to increase during time, and hence the choice of the best one implies to solve a non-stationary problem. We provide a theoretical framework to solve this difficult problem. A first approach to handle the selection of learning experts problem is to reduce it to stochastic problems: the experts are learned in parallel during a first phase, then they are explored and exploited in a second phase. A second approach models the selection of learning experts as an adversarial problem with a finite budget of contaminated rewards. We call this setting adversarial bandits with budget. Here, experts are learned, selected and exploited at the same time. When the budget of contaminated rewards is known, we propose and analyze a randomized variant of the algorithm Successive Elimination. When this budget is unknown, we analyze the algorithm EXP 3 for the proposed setting. We illustrate both approaches in the case where the learning experts are based on Bandit Forests initialized with different sets of parameters.