An Exploratory Study of the Relationship Between Software Test Smells and Fault-Proneness

Test smells have been defined as indicators of poorly designed tests. Their presence negatively affects the maintainability of a test suite as well as the production code. Despite the many studies that address the negative impacts of various test smells, until now there has been no empirical evidence considering the relation between the evolution of test smells and that of faults in the production code. This paper presents such evidence. It presents a case study of data collected from 28 versions of Apache Ant that include a total of 4,447 unit tests. Three key results arise from the data. First, the absolute number of test smells increases as Apache Ant evolves. Second, some test smells are positively correlated with the existence of faults in the production code. Finally, our results show that it is possible to predict faults in the production code based on the existence of test smells in the code’s unit tests. In addition, the resulting prediction model is more accurate at predicting high-severity faults than low-severity faults. This is an important result as it enables an engineer to focus preventative maintenance efforts, applied to the production code, using test smells found in the unit tests.