A Multi-Modal Approach to Measuring the Effect of XAI on Air Traffic Controller Trust During Off-Nominal Runway Exits

Lack of transparency has been demonstrated to be a stumbling block in building Air Traffic Controller (ATCO) trust towards intelligent decision aids. To address this issue, a runway exit prediction decision aid, with explainability, was developed with the trait of providing explanations involving the top three contributing features to its predictions. To evaluate the influence of the intelligent decision aid's explanations on ATCO Trust, the decision aid was used in a human-in-the-loop study during off-nominal runway exits, utilizing 12 participants and a total of 67 trials. A multi modal approach was adopted with three types of data (questionnaire, behavioural, physiological) being collected in this study to ensure a more comprehensive understanding of the effects of explainability on ATCO trust. The results indicated that higher levels of perceived transparency led to an increase in trust levels, with an accompanied increase in cognitive load and complacency, even with low prediction accuracy by the intelligent decision aid. As such, these effects must be accounted for in designing XAI decision aids, which are defined as decision aids that rationalize their recommendations, for off-nominal events, when attempting to enhance trust levels by increasing transparency.