Characterization of MPI Usage on a Production Supercomputer

MPI is the most prominent programming model used in scientific computing today. Despite the importance of MPI, however, how scientific computing applications use it in production is not well understood. This lack of understanding is attributed primarily to the fact that production systems are often wary of incorporating automatic profiling tools that perform such analysis because of concerns about potential performance over-heads. In this study, we used a lightweight profiling tool, called Autoperf, to log the MPI usage characteristics of production applications on a large IBM BG/Q supercomputing system (Mira) and its corresponding development system (Cetus). Autoperf limits the amount of information that it records, in order to keep the overhead to a minimum while still storing enough data to derive useful insights. MPI usage statistics have been collected for over 100K jobs that were run within a two-year period and are analyzed in this paper. The analysis is intended to provide useful insights for MPI developers and network hardware developers for their next generation of improvements and for supercomputing center operators for their next system procurements.