Joint Sleep Control and Energy Sharing Strategy With Deep Reinforcement Learning in Green Ultra-Dense Networks

Ultra-dense network (UDN) is considered a viable solution to meet the growing demand for mobile data traffic. However, the massive deployment of small base stations (SBSs) dramatically increases the energy consumption and carbon footprint of the network, posing significant environmental challenges. In this paper, we design a green UDN system driven by renewable energy sources (RES) that utilizes sleep control and energy sharing methods to reduce grid energy consumption and enhance the utilization of RES. To address the dynamic variations in harvested RES and traffic load arrivals, we propose a joint sleep control and energy sharing strategy based on deep reinforcement learning (DRL). Specifically, we formulate the sleep control and energy sharing problem as a markov decision process (MDP), which can be solved using DRL. To tackle the exponentially growing action space associated with the number of SBSs, we develop an SBS energy management algorithm based on multi-discrete proximal policy optimization (PPO) with a multi-dimensional action space output. Numerical experiments demonstrate that our proposed scheme significantly outperforms existing methods in terms of energy efficiency in green UDNs.