Coverage Metrics to Measure Adequacy of Black-Box Test Suites

In black-box testing, one is interested in creating a suite of tests from requirements that adequately exercise the behavior of a software system without regard to the internal structure of the implementation. In current practice, the adequacy of black-box test suites is inferred by examining coverage on an executable artifact, either source code or a software model. We propose the notion of defining structural coverage metrics directly on high-level formal software requirements. These metrics provide objective, implementation-independent measures of how well a black-box test suite exercises a set of requirements. We focus on structural coverage criteria on requirements formalized as Linear Temporal Logic (LTL) properties and explore how they can be adapted to measure finite test cases. These criteria can also be used to automatically generate a requirements-based test suite. Unlike model or codederived test cases, these tests are immediately traceable to high-level requirements.