A Bayesian Decision Theory Paradigm for Test and Evaluation

Traditional methods for the Test and Evaluation of military systems are based on a combination of ensuring that requirements are met and optimizing the information gained during testing. However, neither approach can address the following fundamental question: how much is a test worth? There are obvious practical benefits to being able to answer this question. Furthermore, the inability of traditional methods to address it suggest that they are not capturing the essence of what the Test and Evaluation process is. This paper presents a new approach to Test and Evaluation based on Bayesian Decision Theory. It maintains the current knowledge one has about the parameters that govern a system’s behavior, updating this knowledge using a Bayesian filter whenever new data arrive. It couples this Bayesian filter with a utility function that is based on the system’s operational utility: i.e., the value of the deployed system to its various stakeholders. In this paradigm, the value of the system to its stakeholders and the cost of testing can be expressed in a common currency of dollars. Therefore it answers the question of how much a test is worth as an organic by-product of how it functions. This paradigm, called Dynamo, is explored in a simple scenario that demonstrates the insights it offers into the nature of Test and Evaluation. It demonstrates that an arbitrary utility function can be decomposed into an intrinsic component that is concerned only with producing the correct decision, and a cost of imprecision which values information about a system for its own sake. Some testing protocols implicitly optimize utility functions that have a negative cost of imprecision, leading to the pathological behavior of declining to test even when testing is free. The paper concludes with a discussion of how Dynamo was implemented to perform a case study on test data from a specific radar system.