Towards Certification of Adaptive Flight Automation Systems: A Performance-Based Approach to Establish Trust

Unforeseen situations in an ever-changing environment are inherent to flight operations. After having mastered the basic flight training exercises, pilots are confronted with various training scenarios. These scenarios are created, inter alia, based on findings from investigations of incidents and accidents in aviation, or on negative operational trends identified in the flight data monitoring of an airline operator. Pilots are expected to cope with uncertainties, build up experience, apply lessons learned from the continuous training and checking, and even exert creativity should a situation require it.When trying to implement a fully autonomous flying system, it becomes evident that traditional rule-based algorithms reach their limits. It is not possible to identify and elaborate the entire input space of realistic — and even unrealistic — situations an electronic pilot might encounter in operation. Furthermore, a specific situation might have to be handled differently if the environmental conditions change, and system failures may happen at any time and even simultaneously, further increasing the extent of the imaginable input space. Thus, future flight automation systems will likely include adaptive algorithms, including machine learning systems with learning-enabled functionality. They could consist of multiple layers of adaptive components interconnected with conventional rule-based algorithms. The resulting architecture with the interdependent components would be rather complex.Such adaptive systems could then build up flight experience — both during their training and once in operation — and adapt their behaviour accordingly, with the goal of improving their performance. They learn from mistakes and identify, analyze and memorize situations where another chain of decisions and actions could have improved the result — to enhance safety, autonomy, and robustness under all conditions — like human pilots. Even beyond, both such positive and negative learnings could be conserved and disseminated to other fellow electronic pilots, exponentially widening the flight experience.However, there are barriers to the deployment of such adaptive systems. The current regulatory framework for certification of civil avionic systems is based on the premise that a system’s correct behaviour must be specified entirely and verified prior to operation, and that its pre-programmed behaviour remains invariant.Current regulatory initiatives towards the certification of adaptive systems address the development process, aiming at establishing a so-called learning assurance for self-learning systems. These initiatives approach the challenge bottom-up, focusing on the development phase.The paper at hand elaborates on a top-down, black-box testing approach emphasizing the idea of thorough verification of an adaptive flight automation system in analogy to the skill tests and proficiency checks human pilots have to pass. This anthropomorphic approach sets a performance-based focus, which assesses the overall output of the adaptive system.The concept of an electronic flight instructor and flight examiner is presented in anticipation of the elevated training needs of machine learning systems compared to human pilots. This concept will be embedded in a research setup for development and validation. These two components shall implement a subset of the training and checking syllabi towards a pilot licence for adaptive flight automation systems.One of the key issues is to define the interface between the flight instructor/examiner and the flight automation system, allowing a seamless integration. Flight tasks have to be generated and communicated to the electronic pilot, and a quantitative performance assessment and feedback must be possible.Finally, a combination of the bottom-up and the top-down approach is outlined, with both methods working in tandem, with intersecting phases of training and learning. Such a combination could build the necessary foundation of trust, a crucial prerequisite for certification, and enable the deployment of adaptive flight automation systems in the future. In effect, this could open up the door to a number of new opportunities in the field of flight automation — while maintaining or even improving the overall level of safety, and while relying on the well-proven and continuously refined regulatory framework for licencing of human pilots.