Real-time Vision-based Control of SWaP-constrained Flight System with Intel Loihi 2

The Neuromorphic based system contained in this research learns from the human pilot. The typical system for this type of research would have been designed for GPUs utilizing a Deep Neural Network. However, previous research indicates that GPUs require relatively large amounts of energy to operate and produce comparatively high amounts of heat. Moreover, due to Low Size Weight and Power (SWaP) constraints for Aircraft (A/C) and for Uncrewed Aerial Systems (UAS), GPUs may have to be off-board and or cloud computing utilized. Which may not be practical in situations. Therefore as an alternative the neuromorphic processor Loihi 2 was chosen. In this research, it was found that Loihi 2 solves these problems while showing to provide exceptional and robust performance. The Lava Sigma-Delta Neural Network (SDNN) inspired by PilotNet can be used in such a way to fly an aircraft or drone by learning from the human pilot using the neuromorphic Loihi 2. This study utilized actual live flight test data the researchers collected from live drone test flights to train an SDNN. During this research, a model was developed using the human hippocampus and visual cortex as they relate to the SDNN. The results of this study should enable an expanded autonomy envelope for A/C and UAS. This research will allow the creation of more scientific and engineering discovery and autonomy applications, ultimately empowering more mission efficacy by way of fast, on the edge, and stand alone computing. Demonstrated by the SDNN was a robust algorithm that emulated biological systems and generalized very well. To the best of our knowledge, this is the first in the world’s work that enables a UAS to be autonomously flown with Loihi 2, utilizing the SDNN.