AoI-Energy-Aware UAV-Assisted Data Collection for IoT Networks: A Deep Reinforcement Learning Method

Thanks to the inherent characteristics of flexible mobility and autonomous operation, unmanned aerial vehicles (UAVs) will inevitably be integrated into 5G/B5G cellular networks to assist remote sensing for real-time assessment and monitoring applications. Most existing UAV-assisted data collection schemes focus on optimizing energy consumption and data collection throughput, which overlook the temporal value of collected data. In this article, we employ Age of Information (AoI) as a performance metric to quantify the temporal correlation among data packets consecutively sampled by the Internet of Things (IoT) devices, and investigate an AoI-energy-aware data collection scheme for UAV-assisted IoT networks. We aim to minimize the weighted sum of expected average AoI, propulsion energy of UAV, and the transmission energy at IoT devices, by jointly optimizing the UAV flight speed, hovering locations, and bandwidth allocation for data collection. Considering the system dynamics, the optimization problem is modeled as a Markov decision process. To cope with the multidimensional action space, we develop a twin-delayed deep deterministic (TD3) policy gradient-based UAV trajectory planning algorithm (TD3-AUTP) by introducing the deep neural network (DNN) for feature extraction. Through simulation results, we demonstrate that our proposed scheme outperforms the deep
$Q$
-network and actor–critic-based algorithms in terms of achievable AoI and energy efficiency.