Link Analysis for Solving Multiple-Access MDPs With Large State Spaces

Wireless communication networks can be well-modeled by Markov Decision Processes (MDPs). While traditional dynamic programming algorithms such as value and policy iteration have lower complexity than brute force strategies, they still suffer from complexity issues for large state spaces. In this paper, the development of moderate complexity algorithms with high performance is sought for wireless network control. To this end, the approximate value function can be computed by projecting the original value function into a lower-dimensional subspace with the careful choice of basis vectors using tools from graph signal processing (GSP). Although GSP theory mainly focuses on undirected graphs, the transition graphs of MDPs for wireless networks are generally directed. For this reason, graph symmetrization based on the co-link method is considered due to its ability to preserve multi-hop dependencies in the graph. Given the multiple-access model under consideration, key properties of the transition probability matrix are exploited to determine the optimal subspace for the approximate value function with low complexity. The numerical results for a multiple-access channel show that the proposed algorithm can find the optimal basis vectors with high accuracy, and furthermore, the approach is robust to changes in system parameters. It is also shown that the projected equation method outperforms the state aggregation technique in producing higher accuracy with a lower runtime complexity.