COTS clusters vs. the Earth simulator: an application study using IMPACT-3D

In 2002, Japan announced the Earth Simulator - a supercomputer based on low-volume vector processors and a custom network - and reported that computational scientists had used it to achieve 14.9 TFLOPS with the IMPACT-3D code, which is written in high performance Fortran (HPF). Of particular interest was that they had achieved this level of performance using a high-level parallel programming model. There has been considerable concern in the U.S. about the appropriateness of its hardware and software investments in super computing technology. To help assess the U.S. strategy of building systems from commodity-off-the-shelf (COTS) components, we explored using a combination of HPF and scalar compiler technology to tailor IMPACT-3D to microprocessor-based supercomputers and evaluated its performance and scalability on the AlphaServer-based Lemieux cluster at the Pittsburgh Supercomputer Center (PSC). On the Earth Simulator, IMPACT-3D achieved 45% of peak performance on 4096 processors; on 1024 processors of PSC's Lemieux, we achieved 17.29% of peak performance.