A decision-analytic stopping rule for validation of commercial software systems

The decision of when to release a software product commercially is not a question of when the software has attained some objectively justifiable degree of correctness. It is, rather, a question of whether the software achieves a reasonable balance among engineering objectives, market demand, customer requirements, and marketing directives of the software organization. We present a rigorous framework for addressing this important decision. Conjugate distributions from statistical decision theory provide an attractive means of modeling the cost and rate of bugs given information acquired during software testing, as well as prior information provided by software engineers about the fidelity of the software before testing begins. In contrast to other methods, the stopping analysis yields a computationally simple rule for deciding when to release a commercial software product based on information revealed to engineers during software testing-complicated numerical procedures are not needed. Our method has the added benefits that it is sequential: it measures explicitly the costs of customer dissatisfaction associated with bugs as well as the costs of declining market position while the testing process continues; and it incorporates a practical framework for cost-criticality assessment that makes sense to professional software developers.