An Experimental Comparison of Four Unit Test Criteria: Mutation, Edge-Pair, All-Uses and Prime Path Coverage

With recent increased expectations for quality, and the growth of agile processes and test driven development, developers are expected to do more and more effective unit testing. Yet, our knowledge of when to use the various unit level test criteria is incomplete. The paper presents results from a comparison of four unit level software testing criteria. Mutation testing, prime path coverage, edge pair coverage, and all-uses testing were compared on two bases: the number of seeded faults found and the number of tests needed to satisfy the criteria. The comparison used a collection of Java classes taken from various sources and hand-seeded faults. Tests were designed and generated mostly by hand with help from tools that compute test requirements and muJava. The findings are that mutation tests detected more faults and the other three criteria were very similar. The paper also presents a secondary measure, a cost benefit ratio, computed as the number of tests needed to detect each fault. Surprisingly, mutation required the fewest number of tests. The paper also discusses some specific faults that were not found and presents analysis for why not.