On the Use of Specification-Based Assertions as Test Oracles

The "oracle problem" is a well-known challenge for software testing. Without some means of automatically computing the correct answer for test cases, testers must instead compute the results by hand, or use a previous version of the software. In this paper, we investigate the feasibility of revealing software faults by augmenting the code with complete, specification-based assertions. Our evaluation method is to (1) develop a formal specification, (2) translate this specification into assertions, (3) inject or identify existing faults, and (4) for each version of the assertion-enhanced system containing a fault, execute it using a set of test inputs and check for assertion violations. Our goal is to determine whether specification-based assertions are a viable method of revealing faults, and to begin to assess the extent to which their cost-effectiveness can be improved. Our evaluation is based on two case studies involving real-world software systems. Our results indicate that specification-based assertions can effectively reveal faults, as long as they adversely affect the program state. We describe techniques that we used for translating high-level specifications into code-level assertions. We also discuss the costs associated with the approach, and potential techniques for reducing these costs.