How Do Automatically Generated Unit Tests Influence Software Maintenance?

Generating unit tests automatically saves time over writing tests manually and can lead to higher code coverage. However, automatically generated tests are usually not based on realistic scenarios, and are therefore generally considered to be less readable. This places a question mark over their practical value: Every time a test fails, a developer has to decide whether this failure has revealed a regression fault in the program under test, or whether the test itself needs to be updated. Does the fact that automatically generated tests are harder to read outweigh the time-savings gained by their automated generation, and render them more of a hindrance than a help for software maintenance? In order to answer this question, we performed an empirical study in which participants were presented with an automatically generated or manually written failing test, and were asked to identify and fix the cause of the failure. Our experiment and two replications resulted in a total of 150 data points based on 75 participants. Whilst maintenance activities take longer when working with automatically generated tests, we found developers to be equally effective with manually written and automatically generated tests. This has implications on how automated test generation is best used in practice, and it indicates a need for research into the generation of more realistic tests.

I. Introduction

Developers can save time and effort by generating unit tests automatically. Automatically generated tests can be integrated into the code base of the program under test just like manually written tests, where they support developers during software maintenance [1]. Good tests should not merely detect undesired modifications of the code by failing, but should also provide guidance to the developer in correcting the undesired modification to make the test pass again. Furthermore, desired code changes may require the test code to be modified in order to avoid spurious test failures. Accordingly, a good test case should not only be sensitive to deviations from the intended behavior, but should also be maintainable in its own right; it should be easy to understand so that it can be readily adapted to changes in the rest of the code base as it evolves.