A Comparative Study of JVM Implementations with SPECjvm2008

SPECjvm2008 is a new benchmark program suite for measuring client-side Java runtime environments. It replaces JVM98, which has been used for the same purpose for more than ten years. It consists of 38 benchmark programs grouped into eleven categories and has wide variety of workloads from computation-intensive kernels to XML file processors. In this paper, we will compare two proprietary Java Virtual Machines (JVMs), HotSpot of Sun Microsystems and JRockit of Oracle, using SPECjvm2008 on three platforms that have CPUs with the same microarchitecture but different clock speed and cache hierarchies. The wide variations of the SPECjvm2008 benchmark categories, together with the differences in hardware configurations of the platforms, reveal the strong and weak points of each JVM implementation. In the composite SPECjvm2008 performance metrics, JRockit performs 19 to 27% better than HotSpot. This is the results of JRockit's outperforming HotSpot in nine out of eleven categories. However, JRockit is quite weak in JVM initialization as it is revealed from the executions of startup.helloworld; the relative performance of JRockit can be as low as 19% of HotSpot. Another remarkable result is, JRockit runs scimark.monte_carlo much faster (up to 285% of HotSpot) which affects the performance metrics of three categories. The relatively higher performances of JRockit on non-startup benchmarks likely to be the differences in number of x86 instructions executed in JVMs, with exceptions in compiler.* benchmarks. In startup.* benchmarks, the performance differences should also be due to the numbers of x86 instructions executed, but their effects widely vary from benchmark to benchmark.