Helping Pull Request Reviewer Recommendation Systems to Focus

The selection of code reviewers for a pull request can impact the quality as well as the speed of the review. In order to have the best experience both for the developer and the reviewers, there is a need for automatic reviewer recommendation systems for pull requests. Although there has been previous research in this area, it was mainly focused on smaller and medium repositories of up to around 200 developers, while larger repositories were rarely targeted. In this paper we evaluate several existing approaches on a set of 8 Microsoft repositories of different sizes, noticing that the average performance of the approaches seemed to decrease with the number of reviewers the repository has. In order to focus the existing approaches only on relevant reviewers, we propose a technique for improving their performance by scoping down the set of candidate reviewers based on multiple filters. We defined several basic filters and determined that 5 out of the 7 tested existing approaches experienced performance improvements of up to 16.24% better precision and 19.66% better recall averaged over all datasets, with the per dataset improvements peaking at 36.63% better precision and 28.63% better recall. Additionally, by combining different basic filters we were able to achieve additional improvements for 6 out of the 7 existing approaches (with a minor improvement for the remaining approach), which on average over all datasets had up to 17.60% better precision and 21.23% better recall, while the per dataset improvements peaked at 43.28% better precision and 30.94% better recall.