Load Balancing for Ultradense Networks: A Deep Reinforcement Learning-Based Approach

In this article, we propose a deep reinforcement learning (DRL)-based mobility load balancing (MLB) algorithm along with a two-layer architecture to solve the large-scale load balancing problem for ultradense networks (UDNs). Our contribution is threefold. First, this article proposes a two-layer architecture to solve the large-scale load balancing problem in a self-organized manner. The proposed architecture can alleviate the global traffic variations by dynamically grouping small cells into self-organized clusters according to their historical loads, and further adapt to local traffic variations through intracluster load balancing afterwards. Second, for the intracluster load balancing, this article proposes an off-policy DRL-based MLB algorithm to autonomously learn the optimal MLB policy under an asynchronous parallel learning framework, without any prior knowledge assumed over the underlying UDN environments. Moreover, the algorithm enables joint exploration with multiple behavior policies, such that the traditional MLB methods can be used to guide the learning process thereby improving the learning efficiency and stability. Third, this article proposes an offline-evaluation-based safeguard mechanism to ensure that the online system can always operate with the optimal and well-trained MLB policy, which not only stabilizes the online performance but also enables the exploration beyond current policies to make full use of machine learning in a safe way. Empirical results verify that the proposed framework outperforms the existing MLB methods in general UDN environments featured with irregular network topologies, coupled interferences, and random user movements, in terms of the load balancing performance.