Caching Placement and Resource Allocation in UAV Networks for URLLC: A Deep Reinforcement Learning Approach

In this paper, we propose a general optimization framework of caching placement and resource allocation in unmanned aerial vehicle (UAV) networks for achieving ultrareliable low-latency communication (URLLC), where a UAV is deployed to provide edge caching services for ground mobile users. To begin with, an optimization problem for the average content downloading delay minimization is formulated, by jointly optimizing the caching placement, trajectory, and transmit power of the UAV, while satisfying the cache capacity, total transmit power, and kinematics constraints of the UAVs. However, there is an unpredictable nature in the environment, such as changes in content requests, locations of the users and UAVs, making it difficult to be solved by traditional optimization methods. In order to address this difficulty, the formulated problem is firstly modeled as a Markov decision process, and then a deep reinforcement learning framework based on dual-clip proximity policy optimization (DC-PPO) algorithm is proposed to solve this problem. The algorithm adaptively searches for caching placement, trajectory, and transmit power of UAV, and a better match between action and state is ensured by using a soft greed strategy to make decisions. Finally, extensive simulation results show that the proposed DC-PPO algorithm outperforms the benchmark algorithms in terms of the average content downloading delay and the convergence speed. Moreover, compared to other schemes, the average content downloading delay is reduced by the proposed joint design scheme.