Challenges in developing an aviation operational performance dashboard

Ongoing measurement of system performance is an essential input to management of the National Airspace System (NAS) by the Federal Aviation Administration (FAA). MITRE has been working with the Air Traffic Management System Command Center (ATCSCC) since 2000 to provide new performance metrics and data visualizations to better measure and understand system performance, and inform changes to behavior to reduce delays in the future. Accurate measurement of the portion of system performance attributable to FAA action is difficult, as several factors outside the FAA's control heavily influence performance metrics, including weather and user traffic demand. So the detection of anomalous performance, either on a particular day, or in a longer term trend, can be challenging. The research described in this paper outlines an approach to develop a set of data visualizations to identify performance anomalies for the FAA, focusing primarily on block time and flown distance. This approach is focused on developing triggers to identify anomalies, both for individual days and in trends. It uses an approach utilizing statistical regression, which will be described in detail. The calculation of the metric and its baseline value will be described in the paper. In the examples which will be documented in the paper, in both the static and dynamic cases, the control limits are based on a 95th percentile of the average daily values during a 2012-2014 baseline period. In the static case, it is based on the statistical distribution of the metric itself (average daily block times above a baseline value) over the baseline time period. In the dynamic case, it is based on the statistical distribution of the residuals from the regression, which models the metric as a function of a daily weather index (ASPM weighted weather index). By using this dynamic approach, a day is triggered as an anomaly for further attention not simply because it is a large value, but because it is larger than the expected normal range for that metric after weather is factored out. In the paper the approach will be extended to include multiple factors, not just weather. All included factors will be those outside the control of the FAA (exogenous). By doing this, the remaining (residual) variation in the metric's behavior, after factoring out the exogenous factors, will be more representative of actual system performance which can be affected by changes in FAA actions. An additional method will also be described to identify anomalies in trends, using a similar statistical approach.