Extendable framework for monitoring heterogeneous multi-accelerator HPC cluster

The superior performance:power ratio of accelerators is motivating new cluster architectures with varied accelerator combinations. Monitoring ensures normal functioning of the cluster by detecting service degradations and prompt rectification. This paper describes a modular and extendable monitoring framework for heterogeneous multi-accelerator clusters which will be useful for future HPC systems. The framework can support third party software plugins to provide different functional features. A monitoring tool has been developed on the basis of this framework to monitor CPU, GPGPU and FPGA accelerators, network, storage, user jobs and other relevant services of a heterogeneous cluster; the tool is also capable of auto rectification to a certain extent.