Flight Trajectory Control With Network-Oriented Hierarchical Reinforcement Learning for UAVs-Assisted Data Time-Sensitive IoT

Within the Internet of Things (IoT) for traffic monitoring, the employment of autonomous aerial vehicles (AAVs) as relays for collecting and transmitting real-time data from traffic sensors to base stations has proven a promising approach. In UAV-assisted Data Time-Sensitive IoT (DTIoT), the Age of Information is a crucial metric assessing data freshness, measuring the elapsed time from traffic sensors to the base station. Optimizing flight trajectories of multiple UAVs to minimize AoI while adhering to energy constraints poses a significant challenge. Current research often employs deep reinforcement learning for UAV trajectory control. Nevertheless, managing multi-agent continuous trajectories in intricate DTIoT network environments faces obstacles due to sparse rewards, thus impeding the training of deep neural network-based control policies using traditional DRL techniques. In this paper, we propose a network-oriented hierarchical reinforcement learning (NO-HRL) to control the UAVs’ flight trajectory in DTIoT networks for minimizing the AoI. We devise a control policy leveraging a two-tier hierarchical DRL framework, with the upper tier determining the target and the lower tier executing it. We also introduce a decoupled sequential training approach to efficiently train the mutually dependent two-tier DRL network of NO-HRL. Experimental results demonstrate that our method excels in optimizing AoI for DTIoT compared to other baselines.