Deep Reinforcement Learning for Energy-Efficient Multi-Channel Transmissions in 5G Cognitive HetNets: Centralized, Decentralized and Transfer Learning Based Solutions

Energy efficiency (EE) constitutes a key target in the deployment of 5G networks, especially due to the increased densification and heterogeneity. In this paper, a Deep Q-Network (DQN) based power control scheme is proposed for improving the system-level EE of two-tier 5G heterogeneous and multi-channel cells. The algorithm aims to maximize the EE of the system by regulating the transmission power of the downlink channels and reconfiguring the user association scheme. To efficiently solve the EE problem, a DQN-based method is established, properly modified to ensure adequate QoS of each user (via defining a demand-driven rewarding system) and near-optimal power adjustment in each transmission link. To directly compare different DQN-based approaches, a centralized (C-DQN), a multi-agent (MA-DQN) and a transfer learning-based (T-DQN) method are deployed to address whether their applicability is beneficial in the 5G HetNets. Results confirmed that DQN-assisted actions could offer enhanced network-wide EE performance, as they balance the trade-off between the power consumption and achieved throughput (in Mbps/Watt). Excessive performance was observed for the MA-DQN approach (>5 Mbps/Watt), since the decentralized learning supports low-dimensional agents to be coordinated with each other through global rewards. In further comparing the T-DQN against MA-DQN solutions, T-DQN presents beneficial usage for very low or very high inter-cell distances, whereas the usage of MA-DQN is preferred (by a factor of ~1.3) for intermediate inter-cell distances (100-600m), where the power savings are feasible towards achieving increased EE. Furthermore, T-DQN scheme guarantees good EE solutions (above 2 Mbps/Watt), even for densely-deployed macro-cells, with effortless training and memory requirements. On the contrary, MA-DQN offers the best EE solutions at the expense of massive training resources and required training time.