Reinforcement Learning Based Dynamic Energy-Saving Algorithm for Three-tier Heterogeneous Networks

To cope with the rapidly growing demand for data traffic, heterogeneous network (HetNet), including different types of base stations (BSs), is advocated as a promising network architecture. Considering the different quality of service (QoS) requirements of the Internet of things (IoT) users and ordinary users, we propose a three-tier HetNet model with non-equal bandwidth. To reduce the power consumption caused by the dense deployment of BSs, we propose a novel reinforcement-learning (RL) based dynamic pico-cell base station (PBS) operation scheme. The proposed RL scheme is based on the asynchronous advantage actor-critic (A3C) algorithm, and can dynamically determine the on/off state of each PBS, aiming to achieve the minimal total power of the macro-cell without any prior information. Simulation results show that the proposed algorithm can achieve 92.1% performance gain of the optimal level in terms of the power consumption saving while needs less training time and lower running resource requirements compared to the benchmarks.