Efficient layering for high speed communication: Fast Messages 2.x

The authors describe their experience designing, implementing, and evaluating two generations of the high performance communication library, Fast Messages (FM) for Myrinet. In FM 1.x, they designed a simple interface and provided guarantees of reliable and in-order delivery, and flow control. While this was a significant improvement over previous systems, it was not enough. Layering MPI atop FM 1.x showed that only about 20% of the FM 1.x bandwidth could be delivered to higher level communication APIs. The second generation communication layer, FM 2.0, addresses the identified problems, providing gather-scatter, interlayer scheduling, receiver flow control, as well as some convenient API features which simplify programming. FM 2.x can deliver 70-90% to higher level APIs such as MPI. This is especially impressive as the absolute bandwidths delivered have increased nearly fourfold to 70 MB/s. They describe general issues encountered in matching two communication layers, and the solutions as embodied in FM 2.x.