Trace-based analysis and tuning for distributed parallel applications

We present an integrated approach to deal with timestamp consistency, and trace based performance analysis techniques for distributed parallel applications. Our trace generation facility captures message passing and system events such as process dispatch with minimal trace overhead. Trace driven analysis tools are developed for post execution analysis, reporting information such as the time stolen by other processes in each node, and the observed message passing time and local wait time for each message. We then present our techniques to reduce total elapsed times based on observed message passing times and local wait times.