Mutation Analysis of Parameterized Unit Tests

Recently parameterized unit testing has emerged as a promising and effective methodology to allow the separation of (1) specifying external, black-box behavior (e.g., assumptions and assertions) by developers and (2) generating and selecting internal, white-box test inputs (i.e., high-code-covering test inputs) by tools. A parameterized unit test (PUT) is simply a test method that takes parameters, specifies assumptions on the parameters, calls the code under test, and specifies assertions. The test effectiveness of PUTs highly depends on the way that they are written by developers. For example, if stronger assumptions are specified, only a smaller scope of test inputs than intended are generated by tools, leading to false negatives in terms of fault detection. If weaker assertions are specified, erroneous states induced by the test execution do not necessarily cause assertion violations, leading to false negatives. Detecting these false negatives is challenging since the insufficiently written PUTs would just pass. In this paper, we propose a novel mutation analysis approach for analyzing PUTs written by developers and identifying likely locations in PUTs for improvement. The proposed approach is a first step towards helping developers write better PUTs in practice.