A Twin-Delayed Deep Deterministic Policy Gradient Approach for UAV Formation Control

This paper explores the use of a Twin-Delayed Deep Deterministic Policy Gradient (TD3) approach for the formation control of multiple Uncrewed Aerial Vehicles (multi-UAVs). A leader-follower configuration is adopted. The proposed TD3 algorithm integrates the Deep Deterministic Policy Gradient (DDPG) algorithm with the double
$Q-\mathbf{learning}$
technique, creating a continuous controller for the formation tracking of multi-UAVs. Unlike the DDPG, TD3 utilizes two
$Q-\mathbf{networks}$
that explore the environment independently, selecting the smaller
$Q-\mathbf{values}$
to compute targets and update the
$Q-\mathbf{functions}$
. Additionally, the target action policy is constrained within a valid action range, and updates are deliberately delayed to prevent the exploitation of erroneous experiences. Simulation results validate the effectiveness of the proposed method.