Selecting V&V Technology Combinations: How to Pick a Winner?

Numerous software verification and validation (V&V) techniques and tools exist to analyse requirements, designs and implementations of software systems. These V&V technologies range from relatively lightweight ones, such as inspection and testing, to more heavyweight technologies based on formal methods and theorem proving. For complex systems, a significant part of the cost and effort for development and maintenance is associated with V&V activities, and this almost always involves selecting and applying a mix of V&V technologies. Unfortunately, little is known about the cost-effectiveness of individual technologies or how to derive the most cost-effective combination. As such, combinations for particular projects are typically selected in an ad-hoc way. In this paper, several issues related to the selection and evaluation of combinations of V&V technologies are explored, based on personal experiences with the V&V of concurrent Java components. A V&V method is presented that was systematically derived through an analysis of the possible failures that can occur in concurrent Java components. This method combines inspection, static analysis, and dynamic testing. In addition, empirical methods that use analysis of fault data and experiments to evaluate V&V combinations are presented. Finally, ideas are presented for an iterative method that can be used to assist with both the selection and evaluation of cost-effective V&V combinations in a given context.