From Sensors to Rotors: Generative Neural Network Driven Hierarchical Learning for Agile Drone Dynamic Target Tracking

Traditional motion planning pipelines for autonomous drones often separate perception, planning, and control into distinct layers, resulting in information loss and reduced agility, particularly in dynamic target tracking within obstacledense environments. End-to-end neural networks offer alternatives but face challenges with data utilization and exploration. This paper introduces a hierarchical learning framework for tracking dynamic targets in complex, cluttered environments, built around a vector quantized variational autoencoder (VQ-VAE) architecture. The framework uniquely enables drones to learn what lowlevel rotor commands achieve expert-demonstrated high-level goals, rather than directly mimicking expert outputs. Our three-stage approach generates high-quality expert trajectories using global SE(3) optimization, employs reinforcement-supervised learning to map perceptions to rotor commands, and uses reinforcement learning with pre-trained VQ-VAE components to surpass initial demonstrations. The VQ-VAE serves as a knowledge bridge between stages, preserving control skills while enabling exploration. Experiments in dense obstacle scenarios with dynamic targets demonstrate sub-second planning cycles regardless of obstacle density, success rates exceeding 90% in moderately complex environments, and velocities 4-5 times higher than comparable approaches, proving the framework's effectiveness for agile navigation in unpredictable, obstacle-rich environments.