A Project-level Investigation of Software Commit Comments and Code Quality

Useful and meaningful commit comments are assumed to contribute to an effective software development process. In fact, some investigations of software quality factors have assumed the existence of this relationship. However, to the best of our knowledge, no research has demonstrated an empirical link between commit comment quality and code quality. To fill this gap, in this work we acquired 120 open source projects from GitHub, examining the relationships between the commit comment quality and code quality over a 6-month period. We first calculated a set of comment quality metrics and a set of code quality metrics. Two of these metrics, comment expressiveness and code thrashing frequency, were newly developed as part of this research. Then we evaluated the relationship between these two sets of metrics. We found strong evidence that the new thrashing frequency metric is a valid measure of code quality. Considering our main hypothesis, we found comment uniqueness and percentage of comments that have bodies, both measures of comment quality, had significant relationships with thrashing frequency in the expected direction, namely that better comments were associated with lower thrashing frequency. Measures related to comment length and comment word frequency, however, produced some significant relationships in the opposite direction, suggesting that developers may write more elaborate comments when they are concerned about introducing defects. Overall, our results indicate that the relationships between comments and code are more complex than suggested by our original hypothesis.