From Specialized Mechanics to Project Butlers: The Usage of Bots in Open Source Software Development

We seek to identify how open source software (OSS) projects adopt bot services from a diverse set of selections. Our empirical research examines bot usage in the most popular OSS repositories in GitHub.

TODAY, MAINSTREAM OPEN source software (OSS) development often involves a huge base of contributors, users, and other stakeholders, which is afforded by various online social coding platforms, e.g., GitHub. 1 At the same time, the fast, ongoing penetration of the continuous integration, delivery, and deployment (CI/CD) model makes urgent demands on rapid iterations. Facing increasing pressures from managing a large number of contributors and their contributions and the ever-faster pace in CI/CD, some small-scale automation techniques have found their way into OSS projects' engineering practices.
Software agents for software engineering (SE) tasks, i.e., developer bots, or bots, for short, are exactly such automation techniques that have the potential to significantly automate many aspects of building software. Various bots have been proposed for improving developers' workflow through better productivity and work-life balance. 2 These software agents and bots automate repetitive and routine tasks, such as managing issues and pull requests, building and executing test suites, and so on. 3,4 Thus, they can often substitute for human labor. While these bots have saved substantial human efforts, they also have exhibited limitations, for instance, not being designed for smart tasks and lacking interactivity during humanbot communication. 3,5 Bots will inevitably be adopted by more and more OSS projects in the foreseeable future of automated SE. Thanks to the continuous efforts of practitioners and researchers, the current bot market offers a wide range of development services 6 for OSS maintainers to shop around. However, we have little knowledge about how practitioners select and adopt these bots.
Here, we start to bridge the gap by empirically examining the bots adopted by popular OSS projects on GitHub, the largest online social coding platform. Combining automated bot detection and manual review, we compile a list of bots used in 1,000 popular projects. Analyses of these bot services show a remarkable adoption rate (61%) among popular GitHub OSS projects when compared with prior studies. 4  in the most popular OSS repositories in GitHub.// application scenarios of these bots are still limited: many of them are combinations of simple automation features, which are triggered by specific project events, and they perform designated tasks through a rule-based mechanism. In addition, over 69% of bots remain private, without transparently sharing their implementation and being accessible outside projects and organizational ecosystems.

Bots in OSS Development
In the past decade, the emerging CI/CD model and the large-scale collaborative software development powered by online social coding platforms have driven the rapid evolution of OSS development practices. Fast iterations drastically complicate the development process as projects evolve. Therefore, practitioners resolve to exploit automation techniques that handle specific routine development and social tasks so that they can focus on innovative activities.
These automation techniques, i.e., bots, may support developers through many service mediums. Some bots have user profiles and are authorized as their projects' contributors. 4 Besides these account-based bots, others may employ platform applications, user settings, and various external services. 3 Practitioners often adopt desired bot services with tradeoffs among performance, privacy, and security. SE practitioners and researchers have put significant effort into bot development. 7 Several bot creation frameworks have been developed to separate bot operation logic from specific platform mechanisms, such as GitHub's WebHook event subscription. For example, Probot allows bot developers to interact with GitHub via an internal event emitter, and therefore, developers may focus on designing bot logic. There are other bot creation frameworks that bring specific engineering benefits, such as OpenBot, Fabric Bot, and Windows Community Tool.
Meanwhile, researchers have collected practitioners' feedback for future improvement. For instance, they find that practitioners expect more smart features in bots' communication and functionality. 4 Erlenhov et al. also emphasize the importance for a bot to be socially competent and bring technical value, especially when it is designed for smarter and more complex tasks. 5 Liu et al.'s survey 3 created seven principles for designing bots that demonstrate the critical challenges in interactions.

Methods
This study extracted activity traces of bot accounts and apps from GitHub repositories and identified bots through automatic bot detection and manual review. The data we sought included the following: • a list of popular software development repositories that have employed bots in CI/CD • descriptions of bot accounts, apps, and services employed by these repositories.

Sampled Repositories
The number of GitHub stars is an indicator of popularity and is also associated with project evolution. 8 Sampling most starred repositories helps to avoid many perils in mining repositories, e.g., projects that received minimal participation. 9 Moreover, we ensured that the sample included only software development projects using the pull request model, which excluded non-SE repositories. Finally, due to the limitations of manual reviewing, e.g., substantial time effort, the sample size was set as 1,000.

Data Collection and Analysis
First, to automatically identify bot activities from GitHub repositories, we adopted a bot identification method called BODEGHA, 10 a machine learning classifier using language patterns and features of comments to detect bot activities. Second, we manually examined the automatically classified bot accounts/services and reviewed their activities for data reliability. We collected activity data from the most recent list of event actors in commits, issues, and pull requests. We also reviewed repository information documents, e.g., readme.md. When we found an event actor that was suspiciously repeating similar activities or demonstrated that the content was automatically generated, we first reviewed its GitHub profile or GitHub Marketplace/app listings for its naming and description. Second, we reviewed the content of this GitHub account's recent comments and activities. Finally, for some accounts that did not display their contributions, we collected their activity records through the GitHub application programming interface.
Two researchers generated the preceding review protocol for examining and identifying bot services and tested it on a random subsample of 100 repositories. The process achieved a 0.645 interrater reliability (Cohen's kappa), suggesting substantial agreement. Any differences were resolved during a discussion session and led to updating the protocol. Finally, three researchers followed the updated protocol and labeled the entire sample.
For each identified bot, we extracted several data items, including the bot's service-providing medium, main functionality, service and source code availability, and design/creation framework, if indicated. In addition, we collected the external reference of FOCUS: BOTS IN SOFTWARE ENGINEERING the bot, including its documentation site, GitHub profile page, and main service medium site. Some sets of bots providing the same service often under one application, such as Codecov IO and Codecov Commenter and Dependabot Preview and Dependabot, were merged into one service. A replication package of the manual review protocol and bot data is available at https://tinyurl.com/33392bp3.

Results
Among the 1,000 sampled repositories, BODEGHA identified bot activities in 462. Following the steps described in the "Data Collection and Analysis" section, our manual review found that 613 sampled repositories employed 201 distinctive bot services, though three of them were reported as deprecated or stopped services.
Based on our data, bots' functions fall into six categories: • CI task assistance: These application scenarios and functions of adopted bot services are similar to the findings in prior research. 4,11 Additionally, in our research, we examine combinations of bot functions. Most prevalent is the sole function for automating CI tasks. Combinations of functions are less prevalent in terms of distinctive bot services (refer to Figure 1 for more details).   We do not include GitHub Action as a typical bot service in this study because it is usually powered by existing as well as customized automation.

FOCUS: BOTS IN SOFTWARE ENGINEERING
Many bots have specialized in multiple tasks and become a butler for a repository. For example, RepoKitteh performs multiple tasks, including checking the format of a pull request, automating tests on a pull request, assigning users to issue and labeling problems, and merging pull requests that passed automatic tests. Some organizations have even customized a private butler bot overseeing their CI/CD practices, e.g., Googlebot and Facebook GitHub Bot in Table 1.
Finally, we list the most prevalent bot services, based on how many projects they appear in, in Table 1, and we spot a few common patterns for their services. Noticeably, these services expect Facebook GitHub Bot to provide a quick setup option with GitHub apps and actions. Pull request comments are the major communication channel for them, in addition to Stale, which is also specialized with issues. Further, these services provide a free installment option to the public, though two services are subject only to their organizational ecosystems.

Discussion
Over 60% of the sampled repositories employed at least one bot to automate routine workflows. The high adoption rate suggests that SE bots have become prevalent in OSS projects. Although bots have not been sophisticated enough to handle advanced tasks in practice, the convenience they provide has outweighed many of their drawbacks, e.g., disruptive notifications. 11 The current practice of OSS development on GitHub has largely become a semiautomated procedure heavily assisted by bots.
However, the popular bots we identified employed similar design mechanisms and provided limited interactivity. For a limited number of OS bots, the implementation was with a rule-based system that subscribed to and acted based on certain repository events and event payloads. For example, when offering automatic issue labeling, Carsonbot required a strictly formatted issue with content in its template. Moreover, the interactivity of the bots has not been improved, for various reasons. 7 The basic mechanism is still to respond and act under predefined input for completing simple, tedious tasks. However, we observe the prevalence of multitasking butlers in popular GitHub repositories, and we anticipate a growing percentage of bots that automate multiple tasks instead of specializing in just one. 5 Finally, for over two-thirds (69%) of the identified bots, their implementations remained opaque to the public. Moreover, more than half (53%) were available and applicable only to specific repositories and organizations. We argue that there are two major reasons for bots to be kept private and closed source, according to our observations. First, there are security and business concerns about bots' functions: bots often have a high level of permission (above write access) in the repositories. Open sourcing a bot may expose the vulnerabilities in a repository's automating workflow, which leads to risks of malicious behaviors. Second, many customized butler bots integrate multiple existing services and adapt to the needs of a specific project or ecosystem. Thus, developers have few motivations to share source code, as many components of such bots have been publicly available. However, without a framework that integrates various bot services, the current situation is far from optimal for the development of bots since many repositories choose to withhold their bots.
T his study leveraged automated and manual reviews to investigate the adoption of bots in OSS development practice on GitHub. We found that bots had become more prevalent in OSS development, as expected. 6 However, somewhat unexpectedly, SE bots had not evolved much beyond atomic application scenarios but simply integrated more functions and multiple tasks. This study challenged our ideas about the future of bots. Namely, one line of future work could be to provide an interactive interface for butler bots while communicating with developers during CI/CD practices. Another line of research could be to develop a mechanism or framework to enable a bot-sharing community and address the concerns of security.
supported, in part, by the Donald Bren School of Information and Computer Sciences, University of California, Irvine. Yi Wang is partially supported by the National Natural Science Foundation of China, under grant 62172049.