Warm-Starting Contextual Bandits Under Latent Reward Scaling

Multi-armed bandits have long been known to enjoy optimal long-term performance, with sub-linear cumulative re-gret bounds standard. Recent developments take the performance of early rounds into consideration by ‘warm-starting’ bandits via incorporating pre-existing information into initialisation. Unfor-tunately, existing warm-start approaches are brittle to differences in the reward distributions between pretraining and deployment phases. This paper considers one such contextual bandit setting, where the same linear relationship relates contexts and rewards in pretraining and deployment phases, but only up to (unknown) constant scaling. A probabilistic model is proposed to capture this novel transfer learning problem, and a simple algorithm is derived as a maximum a posteriori point estimate. We present a regret bound for our method, with empirical evaluation across a range of datasets and against several cold- and warm-start baselines. A real-world motivated experiment on database index selection demonstrates nonlinear modelling via neural network feature embeddings.