Spectrum Efficient Mode Selection and Resource Allocation Optimization for D2D Communication in HetNet: A Multi-Agent Q-Learning Approach

Device-to-Device (D2D) communication has shown great potential as a technology for beyond-fifth-generation (B5G) heterogeneous networks (HetNets) to meet the increasing needs of smart mobile devices. However, the implementation of D2D communications is accompanied by various technical challenges and obstacles related to channel and power allocation. In this paper, a joint mode selection and resource allocation scheme is proposed to improve Spectrum Efficiency (SE) in relay-aided D2D communications underlaying HetNets while guaranteeing the quality of service (QoS) for D2D pairs and Cellular Users (CUs). The joint mode selection and resource allocation problem is formulated as a mixed-integer nonlinear programming (MINLP) problem, which is notably difficult to solve. Therefore, the optimization problem is decomposed into three subproblems: mode selection, channel allocation, and power allocation to tackle the optimization problem. First, a greedy-based mode selection algorithm is proposed to select the best mode (direct mode, relay-aided mode) for D2D pairs. Second, a Multi-Agent Q-Learning (MAQL) algorithm is presented to assign the optimal channel for D2D pairs. Then, based on the allocated optimal channel, a modified MAQL algorithm is introduced to allocate the optimal power for each D2D transmitter. The simulation results show the effectiveness of the proposed approach compared to other schemes and validate the significant increase in system sum data rate and SE achieved by spectrum sharing with cellular communication.