Challenges, Strategies, and Impacts: A Qualitative Study on UI Testing in CI/CD Processes from GitHub Developers' Perspectives

Continuous Integration and Continuous Delivery (CI/CD) processes are vital to meet the growing demands of open source software (OSS), providing a pipeline to enhance project quality and productivity. To ensure the user interfaces (UIs) of these systems work as intended, UI testing is crucial for verifying visual elements of software. Integrating UI tests in CI/CD pipelines should provide fast delivery and comprehensive test coverage. However, there is a gap in understanding how popular UI testing frameworks are adopted within CI/CD workflows-and the effects of this integration on OSS development. Aims: This study aims to explore developers' perceptions of the challenges, strategies, and impacts of incorporating UI testing into CI/CD environments. In particular, we focus on OSS developers utilizing popular web-based UI testing frameworks-such as Selenium, Cypress, and Playwright-and popular CI/CD platforms-including GitHub Actions, Travis CI, CircleCI, and Jenkins-on public GitHub repositories. Method: We conducted an online survey targeting OSS developers (n = 94) from GitHub with experience integrating UI testing frameworks into configuration files for CI/CD platforms. To augment our results, we conducted follow-up interviews ($n$ = 18) to gain insights on the challenges, opportunities, and impacts of integrating UI testing into CI/CD pipelines. Results: Our results indicate adapting testing strategy, flakiness and longer executions are major challenges in integrating UI testing into CI pipelines-negatively impacting development practices. Alternatively, the benefits include support for realistic test cases and increased detection of issues. However, developers lack effective strategies to mitigate the challenges, relying on ad hoc trial-and-error based approaches, such as temporarily removing flaky tests from CI workflows until they are resolved. Conclusion: Our findings provide implications for OSS developers working on or considering including UI tests in CI/CD pipelines. We also motivate future directions for research and tooling to improve UI testing integration in CI/CD workflows.