High Performance Block I/O for Global File System (GFS) with InfiniBand RDMA

State-of-the-art network technology has evolved to 10Gbps. However, TCP's high processing overhead and redundant data copies remain a major bottleneck for applications to fully benefit from such high speed technology. Remote direct memory access (RDMA), as an emerging communication protocol, provides an opportunity for efficient storage system design by virtue of RDMA's semantics. Although RDMA based designs have been proposed to improve network file I/O protocols in several previous works, its benefit for cluster file system block I/O is not clear yet. We propose a new technique - "buffer management delegation", which offloads message buffer management to remote communication party. Using this technique, we design our zero copy RDMA based block transfer scheme for GNBD (global network block device), a block access protocol of Red Hat Global File System, to optimize cluster file system performance over 10Gbps InfiniBand network. We evaluate this new scheme in comparison with our copy based scheme and TCP over the same InfiniBand hardware. The evaluation quantifies the redundant copy impact for both bulk data transfer and file system meta-data operations. The results using open source file system benchmarks and widely used system utilities show that our implementation improves GFSperformance by up to 47% compared with copy based scheme, and by up to 136% compared with TCP