Whittle Index Learning Algorithms for Restless Bandits with Constant Stepsizes

We study the Whittle index learning algorithm with Q-Iearning for restless multi-armed bandits. We first discuss Q-learning algorithm with exploration policies-E-greedy, softmax, e-softmax with constant stepsizes. We extend the study of Q-learning to index learning for single-armed restless bandit. The algorithm of index learning is two-timescale variant of stochastic approximations. On slower timescales, we update index and on faster timescales, we update Q-Iearning by assuming fixed index value. Here, Q-Iearning updates are in asynchronous manner. We study constant stepsizes two timescale stochastic approximations algorithm. Further, we study on index learning with deep Q-networks (DQN) learning and linear function approximations using state-aggregations method. We describe the performance of our algorithms using numerical examples and it illustrates that index learning with Q learning DQN and function approximations learns the Whittle index.