WPress: An Application-Driven Performance Benchmark for Cloud-Based Virtual Machines

Approaching a comprehensive performance benchmark for on-line transaction processing (OLTP) applications in a cloud environment is a challenging task. Fundamental features of clouds, such as the pay-as-you-go pricing model and unknown underlying configuration of the system, are contrary to the basic assumptions of available benchmarks such as TPC-W or RUBiS. In this paper, we introduce a systematic performance benchmark approach for OLTP applications on public clouds that use virtual machines(VMs). We propose WPress benchmark, which is based on the widespread blogging software, WordPress, as a representative OLTP application and implement an open source workload generator. Furthermore, we utilize a CPU micro-benchmark to investigate CPU performance of cloud-based VMs in greater detail. Average response time and total VM cost are the performance metrics measured by WPress. We evaluate small and large instance types of three real-life cloud providers, Amazon EC2, Microsoft Azure and Rackspace cloud. Results imply that Rackspace cloud has better average response times and total VM cost on small instances. However, Microsoft Azure is preferable for large instance type.