A Novel Ensemble Q-Learning Algorithm for Policy Optimization in Large-Scale Networks

Reinforcement learning is a classic tool to solve network control and policy optimization problems in unknown environments. The original Q-learning algorithm suffers from performance and complexity challenges when applied to large networks. For networks which admit Markov decision process (MDP) models, a novel ensemble reinforcement learning method which adapts the original Q-learning is proposed for policy optimization in unknown environments. The learning algorithm is driven through the optimization of the Jensen-Shannon diver-gence (JSD). Multiple Q-learning algorithms are run on distinct, synthetically created, but structurally related Markovian systems in parallel, and their outputs are fused based on an adaptive weighting mechanism. The JSD is analyzed, and the convergence of key statistics and Q-functions are provided. Numerical results for several network classes show that the proposed algorithm can achieve up to 50 % less policy error with up to 40 % less runtime complexity versus other state-of-the-art Q-learning algorithms.