Exploring the Effects of Urgency and Reputation in Code Review An Eye-Tracking Study

The Pull-Based development model, a fundamental mechanism of collaboration in modern software engineering (SE), initiates the code review process when a contributor submits pull requests (PRs) for evaluation. Although the decision to approve or decline PRs is often perceived as grounded in their technical quality, prior research presents a more intricate narrative where both non technical factors-beyond the code itself and technical factors influence the acceptance. This study, uniquely integrating cues of urgency (represented by code priority level) and reputation (represented by the experience level of the code’s author), delves into these biases, leveraging eye-tracking technology to illuminate the cognitive processes underpinning the evaluation of PRs.In an experimentally-controlled study involving 37 participants reviewing Java code PRs, we found that perceived priority level of a PR impacted both the time spent on tasks and the associated cognitive load. Moreover, while participants’ behaviors reflected the influence of the priority level and the author’s experience, they remained largely unaware of these effects on their decision-making, highlighting the critical importance of understanding these implicit biases in code reviews. Interestingly, despite variations in attention when reviewing contributions from novice versus senior authors, there was no discernible difference in acceptance outcomes based on the author’s experience. This study takes the next step toward a better understanding of urgency and reputation in SE and may inform future research about code review platforms and guidelines, code reuse, and automated code generation.CCS CONCEPTS• Software and its engineering $\rightarrow$ Software creation and management.