Metrics and development tools for prognostic algorithms

Prognostic algorithms may be classified anywhere in a continuum between being purely data-driven to entirely model-based. The common thread is that they all serve to predict when a future state or states occur with some degree of confidence. It is therefore feasible to consider a generic test bench for "blackbox" prognostic algorithms that can serve as an objective observer as well as an independent design guide. Specifically, the test bench features described in this paper include the ability to identify statistical correlation coefficients through Monte Carlo simulation and compare/contrast approaches with a generic set of metrics such as bias, stability, accuracy, etc., both in a relative sense and to ground truth (verification) data. As a referee, the test bench is a useful tool for performing trade studies on candidate approaches and brokering performance against requirements. As a design aid, the test bench can be used to investigate the largest "cost" drivers for a given application and provide feedback on which uncertainties must be tightly controlled to ensure decision accuracy. Most importantly, a common interface structure allows the test bench to control a given "blackbox" model executable without knowledge of its intellectual property. The general flavor of this paper and illustrative examples are bias towards structural failure prediction in the aerospace community but the concepts are applicable to any prediction or estimation technique.