A Multi-Strategy Multi-Objective Hierarchical Approach for Energy Management in 5G Networks

Network optimization for improving energy efficiency is a complex process given the dynamic nature, numerous performance indicators, and the plethora of dependencies in a wireless environment. In addition, the variable flexibility, complexity, and the resulting energy savings obtained by different strategies make it a challenging task to select one for a given network scenario. Indeed, it is a combination of approaches that could provide the highest savings. To address these challenges and identify the most suitable strategy for a given network scenario, we propose a Multi-Strategy Multi-Objective Hierarchical Reinforcement Learning (MSMO-HRL) based network energy management framework to model different optimization strategies in a distributed manner. This framework uses distributed Double Deep Q-Networks (DDQN) to learn effective strategy selection and optimization policies from high-dimensional network data. By integrating Advanced Sleep Modes (ASMs) in the existing power models and implementing action-wise experience replay, and dynamic reward assignment, we show the possibility to save 10-20% more energy even under high-load scenarios in contrast to an approach that implements only ASMs.