On the Fence: An Offload Approach to Ordering One-Sided Communication

Partitioned Global Address Space (PGAS) and one-sided communication models allow shared data to be transparently and asynchronously accessed by any process within a parallel computation. In order to ensure that updates are performed in the intended order, the programmer must either use potentially slower ordered communication, or perform operations that order unordered communication, such as a fence in the OpenSHMEM model. Often, implementations of such ordering mechanisms require blocking until pending operations have completed remotely, before allowing new operations to be issued. In this work, we present a new queuing technique for the implementation of one-sided communication ordering that is nonblocking and ensures asynchronous progress for pending communication operations. We describe an implementation of this approach using Portals triggered operations to offload queuing of communication operations across ordering boundaries. By eliminating blocking for ordered communication, this approach is able to provide automatic overlap of communication and computation. We demonstrate the benefit of this technique on several applications and measure performance improvements in the 10%-15% range from allowing computation to progress while ordered communication operations are pending.