Multi-Objective Self-Organization Scheduling of Dynamic AAV-MD Networks via Two Stage Deep Reinforcement Learning

With limited computing and communication resources, it is a big challenge for AAVs (Autonomous Aerial Vehicles) self-organization scheduling to achieve the minimum cost and offload delay at the same time, especially in a fully dynamic scenario with ground MDs (Mobile Devices). We propose a two-stage multi-objective self-organization scheduling architecture using deep reinforcement learning (DRL) to solve this problem, in which every single objective attainment is learned prior to learning multi-objective cooperation. In Stage 1, we decompose the original scheduling problem into two parts for achieving two objectives (minimizing energy consumption and minimizing offloading delay): training the AAV’s flight strategy to reduce energy consumption with the offloading policy fixed, and training the task offloading policy with the AAV’s flight strategy fixed. We derive a new multi-objective policy gradient with a credit function for fine-grained credit assignment. In Stage 2, we innovatively introduce a universal value approximator as a function to bridge policy and value functions across the sub-problems, where the parameters in the function are trained by actor-critic approach using the outputs of Stage 1 as its initial inputs. Our algorithm exactly explores how each sub-problem contributes to the overall objective, avoiding the penalty of random exploration when the other two sub-problems have been well rewarded, which can benefit the optimization process. Experimental results show that our approach has good convergence performance and can efficiently cope with large scale AAVs and MDs.