Self-Organization Scheduling of Dynamic AAV-MD Networks via Three-Layer Hierarchical Reinforcement Learning

Self-organization scheduling is a critic must to support the wide use of autonomous aerial vehicles (AAVs) in many scenarios, where frequent changes of network topology caused by high mobility of both AAVs and mobile devices (MDs) lead to great challenges. We propose a self-organization scheduling architecture of dynamic AAV-MD networks containing three sub-problems, respectively for AAVs’ flight route planning, dynamic network topology control and task offloading decision. Three-layered hierarchical reinforcement learning (HRL) is adopted to solve the corresponding sub-problems. In particular, the idea of ‘learn a priori in advance’ is proposed, where one layer is used to learn the current network topology as a prior and feed it to the other two layers as a constraint. Unlike the traditional way of allowing each AAV or MD to select a coalition without any constraint, our method simplifies scheduling by learning a topology related rule for coalition formation without degrading system performance. More importantly, the proposed three-layer deep hierarchical learning framework is organized with two layers working in parallel and nesting with the other one. Simulation results show that our approach has good convergence performance, and can efficiently cope with high mobility of MDs, but without the need of frequent and large-scale AAV position adjustments.