Least cost testing 2016

Testing a system frequently involves repeated independent identical tests with a pass/fail decision being the result of testing. The number of tests to be done typically is fixed in advance of the start of testing. This is a flawed methodology. Test conduct should depend on the outcome of testing to date. If one is confident of the outcome (pass or fail), one should be prepared to quit early. If the outcome is ambiguous, precisely then should testing continue. The testing approach discussed here adopts a Bayesian formalism and fixes, not the number of tests, but the expected number of tests to be done. In the limit that one expects to do several tens of tests, the optimal stopping strategy to minimize probability of pass/fail decision error is well modeled using a diffusive stochastic process. The stopping strategy in this limiting situation is simply expressed as a dimensionless positive real valued function of a single dimensionless positive real variable. Doing enough sequences of tests that expected numbers of tests become numbers of tests, an improvement in testing strategy can be wrought. For fixed number of tests, probability of decision error is decreased. Fixing probability of decision error, the number of tests required is decreased. The decreases can be dramatic. Though the diffusive test strategy function is a limiting case approximation, a Monte Carlo simulation shows how effective the scheme can be. This approach is effective in many testing situations.