S-MFRL: Spiking Mean Field Reinforcement Learning for Dynamic Resource Allocation of D2D Networks

Device-to-device (D2D) technology has been widely used to address the mobile traffic explosion problem due to its ability of direct communications between proximal devices. In practice, the available spectrum is often limited. With the rapid increase of D2D users and cellular users, the efficiency of resource allocation would be dramatically reduced. To overcome the above limitations, this paper proposes a spiking mean field reinforcement learning framework (S-MFRL) to optimize the resource allocation of D2D networks. Firstly, spiking neural network (SNN) cooperated with deep reinforcement learning is trained for channel selection and power control. Secondly, spatio-temporal backpropagation is adopted to accelerate the SNN training. Thirdly, mean field multi-agent reinforcement learning (MFRL) is applied to approximate interactions among D2D users. By this means, the optimization process of resource allocation becomes tractable as the number of D2D users increases, which solves the problem of exponential growth of user interactions. Two algorithms are implemented under the S-MFRL framework by integrating MFRL into spiking actor-critic (S-AC) and spiking proximal policy optimization (S-PPO), which are named S-MFAC and S-MFPPO, respectively. Experimental results show that our designed S-MFAC and S-MFPPO outperform both AC and PPO in terms of convergence rate, access rate, time-averaged overall throughput, and collision probability. Besides, further simulations have been conducted to verify the effectiveness of the proposed algorithm in the case of larger action space and hundreds of D2D users.