Decentralized Multi-agent Reinforcement Learning for Large-scale Mobile Wireless Sensor Network Control Using Mean Field Games

In this paper, the real-time optimal transmission power control problem is investigated for large-scale mobile wireless sensor networks (MWSN). Controlling large-scale MWSN has two novel challenges, i.e., 1) increasing navigation complexity due to a large number of mobile wireless sensors, and 2) limited energy prohibits peer-to-peer communication between large-scale mobile sensors. To overcome these challenges, the novel mean field game theory is adopted and integrated along with the emerging decentralized reinforcement learning technique. Specifically, the optimal transmission control problem and the optimal navigation problem are formulated as mean field games with two objectives. Then, a novel Actor-Critic-Mass multi-agent reinforcement learning algorithm is developed to learn the decentralized optimal transmission power control and motion control. To learn the decentralized optimal navigation and transmission power control policies, the coupled Hamiltonian-Jacobian-Bellman (HJB) and Fokker-Planck-Kolmogorov (FPK) equations are derived in mean field game formulation. The learned decentralized policies can be guaranteed to converge close to the optimal value, i.e., the Nash Equilibrium, even with large-scale MWSNs in uncertain environments. Finally, the numerical simulations have been provided to demonstrate the effectiveness of the proposed design.