Validating Second-Order Mutation at System Level

Mutation has been recognized to be an effective software testing technique. It is based on the insertion of artificial faults in the system under test (SUT) by means of a set of mutation operators. Different operators can mutate each program statement in several ways, which may produce a huge number of mutants. This leads to very high costs for test case execution and result analysis. Several works have approached techniques for cost reduction in mutation testing, like n-order mutation where each mutant contains n artificial faults instead of one. There are two approaches to n-order mutation: increasing the effectiveness of mutation by searching for good n-order mutants, and decreasing the costs of mutation testing by reducing the mutants set through the combination of the first-order mutants into n-order mutants. This paper is focused on the second approach. However, this second use entails a risk: the possibility of leaving undiscovered faults in the SUT, which may distort the perception of the test suite quality. This paper describes an empirical study of different combination strategies to compose second-order mutants at system level as well as a cost-risk analysis of n-order mutation at system level.