Regression testing in Software as a Service: An industrial case study

Many organizations are moving towards a business model of Software as a Service (SaaS), where customers select and pay for services dynamically via the web. In SaaS, service providers face the challenge of delivering and maintaining high quality software solutions which must continue to work under an enormous number of scenarios; customers can easily subscribe and unsubscribe from services at any point. To date, there has been little research on unique approaches for regression test methodologies for testing in a SaaS environment. In this paper, we present an industrial case study of a regression testing approach to improve test effectiveness and efficiency in SaaS. We model service level use cases from field failures as abstract events and then generate sequences of these for testing to provide a broad coverage of the possible use cases. In subsequent releases of the system we prioritize the tests to improve time to detection of faults in the modified system. We have applied our technique to two releases of a large industrial enterprise level SaaS application and demonstrate that using our approach (1) we could have uncovered escaped faults prior to the system release in both versions of the system; (2) using a priority order we could have improved the efficiency of testing in the first version; and (3) prioritization based on failure history from the first version increases the fault detection rate in the new version, suggesting a correlation between the important sequences in versions that can be leveraged for regression testing.