Multirate Distributed Receding Horizon Reinforcement Learning for Optimal UAV-UGV Formation Control

The coordination of unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs) is valuable in many applications, such as emergency search and rescue, and has received increasing attention in recent years. Given their distinct tasks and dynamic characteristics, the UAV team is typically controlled with higher maneuverability for rapid searching, while the UGV team is operated on roads at lower speeds to ensure stability and performance. This discrepancy naturally results in a multirate control problem, which has not been adequately addressed in previous works. Therefore, we present a multirate distributed receding horizon reinforcement learning (RHRL) framework to solve the optimal UAV-UGV formation control problem on fast and slow time scales. The proposed approach includes a distributed RHRL algorithm operating at a slower time scale for the formation control of UGV teams, and another distributed RHRL algorithm functioning at a faster time scale for the formation control of UAV teams. The state information among homogeneous UAV/UGV agents and heterogeneous agents across different teams are exchanged at different frequencies to balance control performance and communication load. Notably, our approach integrates the receding horizon strategy to enhance learning efficiency and provides theoretical guarantees in multirate distributed RL. Theoretically, learning convergence at different time scales and closed-loop stability are guaranteed. Comparative numerical validations are conducted to demonstrate the effectiveness of our approach in heterogeneous UAV-UGV formation control under different time scales and tasks.