Augmented Reality-Based Drone Pilot Assistance Using Consumer-Grade Hardware

Unmanned Aerial Systems (UASs), also known as drones, are experiencing a surge in usage across industries and among consumers of diverse backgrounds. This rise in adoption introduces challenges for novice pilots. This paper explores how Augmented Reality (AR) can help address these challenges by providing an intuitive visualization of the drone's pose given the pilot's third-person perspective of the aircraft. We (1) experimentally evaluate if commercially available AR and UAS technology provides the necessary foundation in terms of data and capabilities, (2) propose a digital twin-based assistance system, incorporating a simulation environment, a Microsoft HoloLens 2, and a DJI Mavic Mini drone to enhance the piloting experience and increase safety by visualizing flight information in real time, and (3) implement the system and perform experiments for a first evaluation of the proposed approach.
^1
1
A commented mixed reality capture video of the final system setup in action is available online at https://youtu.be/R3mcoBQ4NZc.