Trajectory Optimization for UAV-Assisted Communications Based on Hierarchical Reinforcement Learning

Unmanned Aerial Vehicles (UAVs) possess high maneuverability and wide viewing angles, rendering them ideal as flying base stations for resource-constrained Internet of Things (IoT) sensors. For real-time information acquisition and sustainable energy support for numerous IoT devices, an appropriate number of UAVs are required to efficiently deployed for data and energy transfer tasks. However, existing methods face challenges in minimizing the average Age of Information (AoI) due to the complex coupling between trajectory planning and transmission scheduling decisions and the need for efficient coordination in resource-constrained UAV networks. These domain-specific challenges require specialized solutions that effectively balance information freshness and energy efficiency. To address these challenges, we first decompose the scheduling problem into two subproblems: trajectory optimization and transmission optimization. Based on this decomposition, we propose a Hierarchical Trajectory Optimization and Transmission Scheduling (HTOTS) algorithm based on hierarchical reinforcement learning. The HTOTS algorithm employs Deep Reinforcement Learning to sense environmental states in real-time and dynamically adjusts UAV flight trajectories and information acquisition, ensuring an effective balance between data and energy transfer. These subproblems are solved alternately through hierarchical reinforcement learning, which significantly reduces the complexity of each subproblem and improves convergence efficiency. Simulation results show that the proposed HTOTS algorithm outperforms existing methods in terms of average AoI and energy efficiency for various network scales and energy constraints.