Generating expected results for automated black-box testing

In this paper we describe a technique for generating expected results for automated black-box testing. Generating expected results allows larger automated test suites to be created, moving us toward continuous product testing. Our technique uses a program's Input-Output (IO) relationships to identify unique combinations of program inputs that influence program outputs. With this information, a small set of test cases is executed and checked for correctness. Given the correctness of this set, the expected results for the larger combinatorial test set can be generated automatically. Included in the paper is an experimental study in which checking the results of 384 test cases allows us to generate expected results and fully automate nearly 600,000 test cases.