Recognizing User Proficiency in Piloting Small Unmanned Aerial Vehicles (sUAV)

User proficiency in manual operation of autonomous systems is crucial to the performance of these systems because users are often the final barrier in detecting and correcting abnormal behavior in autonomy. This letter presents a new approach to identifying user proficiency in piloting small unmanned aerial vehicles (sUAVs) by first extracting meaningful features and then using a clustering method to generate ground truth. Pilot performance has been broadly explored in the field of aviation, but not for operation of sUAVs, and both are inherently different due expectations for training, location of user, and subsequent change in user’s point-of-view. We propose a novel, hybrid approach to evaluate UAV pilot performance: combining human-rater data and computational methods that incorporate performance metrics to tune and homogenize the process (of applying controls) and the product (UAV flight path) of piloting sUAVs. The results reveal a spectrum of user skills that designers of these systems need to account for and the ways that users at different skill levels can be expected to respond, informing future autonomy design. In a 20 participant study, users were asked to fly a sUAV along 8 different flight paths of varying difficulty while the flight trajectory and user control inputs were recorded. We utilized unsupervised learning techniques to group pilots into proficiency groups and analyzed the clusters with respect to the features built. We also identified possible factors based on groupings to target training of these users. We validated our approach using new set of data from 12 participants.