Identifying Optimization Opportunities Using Memory Access Tracing in OpenSHMEM Runtimes with the TAU Performance System

Exascale systems are expected to have fewer bytes of memory per core available than present petascale systems. Previous analysis of the Open MPI OpenSHMEM runtime has shown that it allocates some object types which use memory proportional to the square of the number of PEs. In this paper, we describe a technique for tracking memory accesses by OpenSHMEM runtimes. By tracking reads and writes, we can determine which object types are constant after initialization; those objects could be shared across PEs which are co-located on the same physical node. This tracking is implemented in the TAU Performance System by using user-level page fault handling in Linux.