Automated Insights Into GitHub Collaboration Dynamics

Today, GitHub (Trademark) is the most widely used platform for open-source software development. Large projects may comprise hundreds of distributed collaborators and thousands of GitHub “issues” (structured discussions). It includes basic support for dealing with issues, via “Pull Requests” (PRs)—document changes that can manually be defined to “close” them (i.e., they address and thereby conclude the issue discussion). Unresolved issues can pile up. For example, at the time of writing, the Kubernetes repository has almost 2000 open issues; finding which ones a PR might close is a hard task by itself. We address this by automatically identifying issue-PR relationships using language models (LMs) and leveraging Information Retrieval (IR) techniques. To foster further research, we contribute a carefully curated novel dataset called CodeConvo, reflecting the most influential open-source repositories for code development as well as some technical document repositories. We use this dataset to benchmark several state-of-the-art (SOTA) non-proprietary models that show exceptional performance on the MTEB benchmark, as well as to train and evaluate the performance of Smart Insights into GitHub Issue-PR Relations (SIGIR), our tailored model. The best SIGIR model/data combination yields an average Mean Reciprocal Rank (MRR) above 0.7, around 20% higher than the best baseline performance. Notably, ablation studies revealed that knowledge transfer occurs not only between different programming languages but also between code and technical documents, albeit to a lesser extent. We believe these results are encouraging and can stimulate the practical application of LMs for taming the complexity of very large projects, in GitHub and beyond.