Federated Reinforcement Learning for Intelligent Route Planning in Aerial-Terrestrial Network

Aerial-terrestrial network (ATN) framework is currently the dominant method of the internet of unmanned agents (IUA) for integrating both aerial vehicles and terrestrial sensors. In ATN scenarios, existing work shows that route planning utilizing deep reinforcement learning (DRL) is important as it can conserve energy for aerial vehicles or diminish network latency. However, utilizing DRL approaches in a multi-agent ATN environment may lead to inefficiencies when sharing original interaction data. In addition, direct data exchange between agents can raise significant privacy concerns. To address these challenges, this work develops a novel federated reinforcement learning (FRL) approach called FRIGE, which takes advantage of ensemble learning for DRL-based methods. Specifically, this paper develops a prioritized gradient ensemble technique for local DRL agents, which constructs twin networks to obtain prioritized gradients without incurring additional interactive costs. After aggregating local models on the server, the latest iteration of the global model is used to update the networks of both agents and their twin networks for subsequent rounds of training. Extensive experiments are conducted on two typical ATN route planning tasks to validate FRIGE’s advancement and generalization capabilities. The numerical results demonstrate that FRIGE not only enhances data sharing efficiency among all agents but also maintains privacy while delivering superior solutions for ATN route planning tasks.