Automating Web Application Testing from the Ground Up: Experiences and Lessons Learned in an Industrial Setting

Automating web application testing is a very tricky process due to inherent complexity, dynamic behavior(s) in web pages, differences in the way browsers render the same content (especially on different form factors), and so on. Yet manual testing is not a practical option, and such automation is a must in the interests of effectiveness and efficiency, because of the large number of browsers/devices that users can choose from, and also given the rapid software development cycles of today. This paper discusses our efforts at Varidesk to automate web tests against our main website - which offers many features and content, but is also a true eCommerce site where users from around the globe can purchase a very broad variety of active workspace solutions that we offer. Our solution was developed in-house, from the ground up, and leveraged and extended freely available automation and test libraries such as Selenium WebDriver and NUnit respectively. We talk about the challenges we faced and how we overcame them, as well as provide technical insights on real-world concerns such as managing test brittleness, and integrating the web tests into an existing Continuous Integration and Continuous Deployment (CI/CD) pipeline. Part of the novelty of this paper is that we are also transparent on the rationale behind our decision to build versus buy, and how we managed resources, especially in terms of cost. We also present lessons learned, and encouraged by the success that we have observed, hope that the results will be beneficial to academia and practitioners alike.

I. Introduction

Automated testing has received a lot of attention over the last few decades, especially to keep up with the rapid pace at which software applications are developed and deployed today. This has extended to automating web application testing as well, which offers its own unique set of challenges. Examples of such challenges are having to automatically test on multiple Browsers (and possibly multiple versions of each Browser); testing on different devices/form factors (where mobile views may differ from standard desktop views); being able to reuse test cases, or partial test scenarios, etc. To help address these challenges, tool support for web test automation has come a long way. For example, to facilitate easy test case creation without writing code, a popular option is to employ tools that allow for record and replay¹

Such a tool first captures inputs and actions (mouse clicks, keyboard entries, navigation commands, etc.) that occur as a web application is utilized. During playback, these inputs and actions are re-delivered to the browser engine.

such as Selenium IDE [19], which is free and open-source. In cases where writing code or scripts, while still leveraging some external tooling, is a viable option, several open-source frameworks also exist such as Selenium WebDriver [20], Appium [1], and CasperJS [6], just to name a few. To facilitate cross-browser execution of tests across a multitude of devices, offerings such as BrowserStack [5] and SauceLabs [18] exist, which allow for users to not have to maintain their own infrastructure by leveraging hosted device farms.