On the Use of GPT to Reveal Common Questions in Developers’ Discussions

Open software development platforms, such as GitHub, foster developers’ collaboration in coding tasks through pull requests (PR). PRs serve as a mechanism for code contributions and structured discussions. These discussions often involve developers exchanging messages that contain questions. Despite the importance of questions, existing studies have not gathered insights on what are recurring types of questions in GitHub PR discussions. In this paper, we address this gap by leveraging the well-established 5W2H framework, which consists of seven questions classes. Our goal is to analyze the distribution of these question types within GitHub PRs and assess whether language models, specifically GPT-3.5 and GPT-4o, can accurately classify them. We conducted our study using a dataset derived from nine GitHub projects from the Netflix and Google ecosystems. Our findings reveal that both GPT-3.5 and GPT-4o perform well in identifying What and Who classes, which were the most frequently occurring question types in developers’ discussions. However, both models struggled with less common categories, failing to detect How Much, and showing inconsistencies in classifying Where, When, Why, and How. These findings suggest that improving classification accuracy may require incorporating domain-specific context and refining prompt engineering techniques.