It takes a village: Monitoring the blue waters supercomputer

The performance of science applications on modern HPC equipment depends on many factors. Architectural features, individual hardware characteristics, and scheduler traits all have an impact on how a particular application performs, not only in isolation but when run in concert with other user applications. Being able to correlate system events and conditions at particular times can give insight into causes of good or bad performance. Unfortunately, the information we seek is not necessarily in a readily accessible form. The problem at hand is how to enable efficient query of the raw data and flexible graphical representation of the results. Web applications that access an underlying database serve this sort of functionality for many science applications quite well. Our scenario of data access is not very different. The data collected for a large HPC environment is complex and grows in size with time. This aspect is different from applications that deal with more static data. It is the dynamic nature of the data that make the problem interesting. In this work we present our approach for the analysis and visualization of HPC system performance data based on database access and web based graphical presentation. We discuss the details of how data is collected and processed from raw logs into the database, how queries are formulated, and how the data are graphically displayed. This process includes dynamic formulation of the queries. Finally we discuss how the system is utilized to analyze system performance.