Delay-aware massive random access for machine-type communications via hierarchical stochastic learning

In this paper, we study the delay-aware access control of massive random access for machine-type communications (MTC). We model this stochastic optimization problem as an infinite horizon average cost Markov decision process. To deal with the distributive requirement and the exponential computational complexity, we first exploit the property of successful access probability to transform the coupling to the constraint on the number of MTC devices attempting to access. As a result, we decompose the Bellman equation into multiple fixed point equations for each MTC device by primal-dual decomposition. Based on the equivalent per-MTC fixed point equations, we propose the online hierarchical stochastic learning algorithm to estimate the local Q-factors and determine the access decision at the MTC devices separately with the assistance of the base station which broadcasts common control information only. Finally, the simulation result shows that the proposed hierarchical stochastic learning algorithm has significant performance gain over the baseline algorithm.