Whittle Index Based User Association in Dense Millimeter Wave Networks

We address the problem of user association in a dense millimeter wave (mmWave) network, in which each arriving user comes with a file containing a random number of packets and each time slot is divided into multiple mini-slots. This problem is an instance of the restless multi-armed bandit problem, and is provably hard to solve. Using a technique introduced by Whittle, we relax the hard per-stage constraint that each arriving user must be associated with exactly one mmWave base station (mBS) to a long-run constraint and then use the Lagrangian multiplier technique to convert the problem into an unconstrained problem. This decouples the process governing the system into separate Markov Decision Processes, each corresponding to a different mBS. We prove that the problem is Whittle indexable, present a scheme for computing the Whittle indices of different mBSs, and propose an association scheme under which, each arriving user is associated with the mBS having the smallest value of the Whittle index. Using extensive simulations, we show that the proposed Whittle index based scheme outperforms several user association schemes proposed in prior work in terms of various performance metrics such as average cost, delay, throughput, and Jain's fairness index. We also compare the proposed policy with the exhaustive search algorithm via simulations, and our results show that the performance of the proposed Whittle index based policy is close to optimal.