Deep-Reinforcement-Learning-Based Proportional Fair Scheduling Control Scheme for Underlay D2D Communication

In the last few years, we have witnessed the usage of billions of Internet-of-Things (IoT)-enabled devices in different applications starting from e-healthcare, transportation, agriculture, etc., across the globe. These interconnected devices share information using the Internet to improve the Quality of Service of the end users. There is a requirement of synchronization among the devices to provide scalability, reliability, and connectivity. Despite these advantages, proximity gain, interference, and fairness are various challenges for these devices in IoT which need to be resolved. To overcome these issues, we propose deep reinforcement learning (DRL)-based control scheme in the underlay of device-to-device (D2D) communication. D2D communication reuses the spectrum resources with cellular user equipment (CUE) to improve spectral efficiency. We propose the joint resource block (RB) scheduling and power control scheme to improve the sum rate of the network while considering the users' fairness among all the links. To solve this problem, first, we transform the nonconvex optimization problem into a multiagent reinforcement learning formulation using the Markov decision process (MDP). Then, to solve the RB allocation, we used the multiagent deep Q-network (DQN) framework to reduce the output dimension and improve the learning efficiency. Then, to convert the stochastic policy into deterministic policy, and to improve the fairness we combine the DQN with deep deterministic policy gradient to form the distributed deep deterministic policy gradient (DDDPG) scheme. Finally, to control the power of both the CUEs and D2D transmitters (DTs), we integrated the conventional optimization scheme with the DDDPG (CO-DDDPG). This combination enhances the convergence speed and reduces the computational complexity of the overall network. Numerical results show that the proposed scheme improves the network sum rate of 11.76% and the fairness 4.21% as compared to the state-of-the-art existing distributed DRL schemes.