An Improved DDPG Algorithm for UAV Navigation in Large-Scale Complex Environments

Autonomous navigation is one of the most critical elementary skills to control UAV navigate around the environment without colliding with obstacles. As UAVs are prevalently applied in many domains, such as goods delivery, search and rescue and intelligent transportation etc, safety and efficiency are two basic requirements while executing tasks. However, autonomous navigation is not a trivial task because it is challenging for UAV to observe, orientate, decide and take actions simultaneously. Specially, in large-scale complex environments, the constrained narrow passages, dense obstacles and dynamic objects pose increasing difficulty. In this article, UAV autonomous navigation is modeled as a Markov decision process and an Improved Deep Deterministic Policy Gradient (ImDDPG) algorithm is proposed to make safe and efficient navigation decisions. In ImDDPG, the actor network is decomposed into two sub-networks to learn a more stable action policy. Next, some reward reshaping functions are developed to solve the sparse reward problem. Then, the convergence speed is further accelerated with a dynamic decay strategy. The targeting neural network is well designed and trained. Simulation experiments are conducted to prove that the proposed ImDDPG algorithm can achieve a success rate more than 95% in large-scale complex environments. Moreover, numerous simulation results are also presented to demonstrate that ImDDPG has better generalization ability than the benchmark DDPG and TD3 algorithms in environments with denser obstacles and dynamic objects.