Multi-Agent Reinforcement Learning for UAVs 3D Trajectory Designing and Mobile Ground Users Scheduling with No-Fly Zones

Unmanned aerial vehicle (UAV)-based aerial communication is considered a promising technology in future wireless systems. In this paper, we study a multi-UAV-assisted data transmission system in an urban environment, where a set of UAVs collect data from mobile ground users (GUs). We provide a design aiming to minimize the total data transmission time by jointly optimizing mobile GUs’ scheduling and the UAVs’ three-dimensional (3D) trajectory while satisfying the requirements of no-fly zones and collision avoidance. The formulated mixed-integer non-convex problem is difficult to address by utilizing traditional approaches, e.g., graph theory and successive convex approximation (SCA), due to the impacts of random GUs moving behaviors and the unpredictable UAV-GU channels. To tackle such challenges, we first transform the joint optimization problem into a Markov decision process. Then a joint optimizing scheme is proposed, including a multi-agent multi-step dueling double deep Q learning network (MAMD3QN) method for UAVs trajectory design and a greedy policy for mobile GUs scheduling. In particular, an improved DDQN network is utilized to optimize UAVs trajectory with dueling networks architecture and multi-step bootstrapping technique. Finally, simulation results show that the proposed design significantly outperforms the benchmark schemes, showcases the advantages of 3D trajectory design over two-dimensional (2D) cases, and highlights the robustness in terms of different NFZs and the mobility of GUs.