What Are You Paying For? Performance Benchmarking for Infrastructure-as-a-Service Offerings

As part of the Cloud Computing stack, Infrastructure-as-a-Service (IaaS) offerings become more and more widespread. They allow users to deploy and run virtual machines in remote data centers (the Cloud), paying by use. However, performance specifications for virtual machines provided by providers are not coherent and sometimes not even sufficient to predict the actual performance of a deployment. To measure hardware performance, hardware benchmarks are available. For measuring the performance of virtual machines in IaaS offerings, these benchmarks are not sufficient, as they don't take into account the IaaS provisioning model where the host hardware is unknown and can change. We have designed a new performance measuring method for Infrastructure-as-a-Service offerings. The method takes into account the type of service running in a virtual machine. By using the method, the actual performance of the virtual machines running a specific IaaS service is measured. This measurement can be used to better compare prices between different providers, but also to evaluate the performance actually available on a certain IaaS platform. We have evaluated the method on several Cloud infrastructure offerings of the Amazon EC2 platform, Flexi scale platform and Rack space platform to validate its utility. We show that already on EC2, the performance indicators given by providers, namely Amazon's Elastic Compute Unit, are not sufficient to determine the actual performance of a virtual machine.