MARL Based UAVs’ Trajectory and Beamforming Optimization for ISAC System

A multiple unmanned aerial vehicles (UAVs) enabled integrated sensing and communication (ISAC) system is investigated. In contrast to existing UAV-enabled ISAC systems assuming static users or 2D UAV trajectory, we consider a practical roaming user scenario and a 3D deployment for UAVs. Then, a joint trajectory and beamforming optimization problem is formulated for maximizing the long-term sum data rate, subject to the transmitting power constraint and ensuring beam pattern gain constraint for sensing target. To address the challenge caused by the dynamic and high dimensionality features, multi-agent reinforcement learning (MARL) is employed for this partial observation Markov decision process (POMDP) problem. We proposed a two-step approach for against the dynamic scenario: 1) a K-means based hierarchical user association algorithm is proposed to renew the user association periodically. 2) a hybrid reward multi-agent proximal policy optimization (HR-MAPPO) algorithm is proposed, which decomposes the complex combined reward into a team reward and an individual reward. HR-MAPPO introduces a hyper-parameter to control the proportion of team/individual action. Numerical results demonstrate that the proposed HR-MAPPO algorithm can outperform conventional single-agent and multi-agent RL algorithms by maintaining high scores on both sum data rate and beam pattern gain.