Performance of MPI Sends of Non-Contiguous Data

We present an experimental investigation of the performance of MPI derived datatypes. While solutions with hardware support can achieve high performance, we show that vanilla approaches fall short. For messages up to the megabyte range most schemes perform comparably to each other and to manual copying into a regular send buffer. However, for large messages the internal buffering of MPI causes marked differences in efficiency. We show how schemes that acknowledge the importance of buffer management give both the highest performance and a simple usage.