Mutation Coverage is Not Strongly Correlated with Mutation Coverage

Several metrics have been proposed in the past to quantify the effectiveness of a test suite; they are usually types of coverage metrics, because it is sensible to quantify the effectiveness of a test suite by the extent to which it exercises (covers) various syntactic or semantic features of a program. Though no coverage metric has emerged as the gold standard of test suite effectiveness, mutation coverage is usually considered as a reliable measure thereof, because the ability of a test suite to detect program mutations can be an indication of its ability to detect faults. In this paper, we aim to challenge the superiority of mutation coverage, by showing empirically that the same test suite can have vastly different values of mutation coverage depending on the mutation operators that are used to generate mutants.