Open-source Code Repository Attributes Predict Impact of Computer Science Research

With an increased importance of transparency and reproducibility in computer science research, it has become common to publicly release open-source repositories that contain the code, data, and documentation alongside a publication. We study the relationship between transparency of a publication (as represented by the attributes of its open-source repository) and its scientific impact (as represented by paper citations). Using the Mann-Whitney test and Cliff’s delta, we observed a statistically significant difference in citations between papers with and without an associated open-source repository. We also observed a statistically significant correlation $(p_{\lt} 0.01)$ between citations and several repository interaction features: Stars, Forks, Subscribers and Issues. Finally, using timeseries features of repository growth (Stars), we trained a classifier to predict whether a paper would be highly cited (top 10%) with cross-validated AUROC of 0.8 and AUPRC of 0.65. Our results provide evidence that those who make sustained efforts in making their works transparent also tend to have a higher scientific impact. CCS CONCEPTS• Applied computing → Publishing; Digital libraries and archives; Digital libraries and archives.