On energy efficiency of liquid cooled HPC datacenters

HPC Data Center's performance and growth are now being limited by both cost and power. A cost-efficient data center and an energy-efficient data center are all too often mutually exclusive, but they do not have to be. Liquid cooling is one area that, when done right, can improve both costs and energy efficiency. The design for liquid cooling systems generally begins with the ASHRAE liquid cooling datacenter classes. These provide guidance to both datacenter facility cooling system designers and electronic equipment manufacturers by providing a common baseline and understanding of the interface conditions between the cooling and the IT equipment. Further, the liquid cooling classes also suggest possible cooling equipment for a given datacenter class. Due to the aforementioned cooling equipment prescription, perception exists that moving from W1/W2 class environments to W3 or W4 classes represent increased energy efficiency during IT equipment operation. In this paper we show this not to be the case universally and explore a more detailed, technical approach to optimizing both cost and energy efficiency. The range of parameters includes geographical and climate conditions, state of the existing data center cooling infrastructure (greenfield, retrofit, cluster change-out, expansion), and IT level liquid cooling architecture. Through this analysis we show that for energy efficient operation of the IT equipment there exists an optimum liquid operating temperature that can also provide the lowest TCO. This temperature can drive the right capital investment as well as reduce facility operational expense and IT operational expense. We also explore the impact on reliability, the controls architecture, use of efficiency metrics, cluster compute performance, and opportunities for energy re-use.