On optimizing I/O through InfiniBand RDMA for commodity clusters

Our goal was to enable pNFS as a highperformance parallel file system by using network file system (NFS) storage objects and InfiniBand remote direct memory access (RDMA) transport in the Linux mainline. One obstacle to this approach was the performance bottleneck in NFS/RDMA streaming-writes from the compute nodes. We benchmarked, tuned, and improved the streaming-write efficiency of the Linux NFS client. However, deeper analyses of the benchmarks and the various I/O short-circuit schemes established upper bounds on the performance of the NFS client — even with an infinitely fast network — so that the performance was substantially less than the theoretical streaming bandwidth of the fast interconnections. The complex interactions between the Linux virtual file system, Linux virtual memory management, and the IB network subsystems apparently impose a limit on further improvement.