Assessing QoS consistency in cloud-based software-as-a-service deployments

Cloud-deployed Software-as-a-Service (SaaS) solutions have become a common global software deployment regime. For SaaS providers success is increasingly tied to social media feedback, customer reviews, and referrals. As such, ensuring sufficiently few users experience low (or poor) quality of service (QoS) levels has become an important concern. Cloud-based SaaS QoS is primarily driven by: i) the incoming workload's pace and complexity, ii) the SaaS system's design and implementation, and iii) the cloud platform's own induced QoS variabilities. Of these, SaaS software engineers generally have the least control over (iii), making it important to properly understand and quantify. This work empirically assesses (iii) by applying statistically rigorous QoS testing to an industry-held cloud-deployed SaaS system. Identical SaaS system instances are instantiated into the same commercial cloud platform and exercised via identical synthetic in-coming workloads. The resulting run-time QoS statistical distributions of each SaaS instance are then pairwise compared via distribution-free goodness-of-fit tests. A high degree of (iii) induced statistical dissimilarity is observed, suggesting significant care is required when seeking to make QoS envelope predictions from per-instance observed SaaS QoS results. This also suggests deeper more formal efforts may be required to better understand and characterize the cloud-induced SaaS QoS consistency issues that arise within modern SaaS deployments.