Testing Artificial Intelligence in High-Performance, Tactical Aircraft

The United States Air Force Test Pilot School (USAF TPS), in partnership with Lockheed Martin Skunk Works, developed and tested a suite of algorithms and artificial intelligence (AI) agents flown on an F-16D. The system was designed to autonomously locate and navigate to an emitter and optically recognize its associated vehicle under the project Have SPI-DERs. This project was the first to explore the use of AI agents trained in simulation using deep reinforcement learning and tested on a high-performance aircraft in the real world performing routing and computer vision tasks. During flight test, the Have SPIDERS project set out to demonstrate the SUT's ability to find and fix the emitter through three phases. The first phase passively detected the emitter using a small antenna array and routed the aircraft using a neural network to reduce the uncertainty volume of the emitter's location. The second phase routed the aircraft using a second neural network to an image point where the emitter could be optically identified. The final phase used a third neural network to optically recognize a vehicle collocated with the emitter. The first two phases each used two feedforward neural networks functioning as actor-critic algorithms. The third phase used a RetinaNet architecture. This paper uses the Have SPIDERS project as a vehicle to discuss lessons learned that can be applied to future AI flight test. These lessons focus on six areas for consideration during system development, test planning, and execution: understanding system under test (SUT) development, instrumenting the AI to understand its behavior, planning for unpredictability, simulation-to-real-world generalization, hardware choices, and manned surrogate testing considerations.