Quantifying Faultiness: What Does It Mean to Have N Faults?

It is common to see software testing experiments where a benchmark program is seeded with N benchmark mutations, then declared to have N faults. We argue that in the absence of a formal definition of what is a fault, let alone what are N faults, it is difficult to assign a meaning to such claims. We further argue that in order to assign a meaning to such a claim we must refer to five parameters, which we introduce and justify. To illustrate our ideas, we consider the tcas component of the Siemens benchmark, which we seed with eight mutations, and we find that not only is it wrong to claim that tcas has eight faults, it is also wrong to measure faultiness by the number of faults in a program.