BT-Duper: A Binomial-Tree Based Data Replication Offloading Method with Native RDMA Primitives

Remote Direct Memory Access (RDMA) has been widely used in distributed storage systems as a low latency network technology, especially for the application of data replication. For the methods using RDMA’s two-sided or one-sided primitives, remote CPU is required to poll the incoming data from replica nodes for data integrity. Alternatively, offloading methods can offload the task of data replication to the RNIC without CPU polling. However, these methods are primarily based on chain replication, including the issues of driver modification requirements and the associated higher average and tail latency. In this paper, we propose an approach based on binomial tree topology to offload the process of data replication to the RNIC with native RDMA primitives, called as BT-Duper. BT-Duper has fewer communication steps compared to the chain replication method, thus achieving lower average and tail latency. Moreover, offloading data replication to the RNIC can only use native RDMA primitives, without modifying the driver or adding additional hardware. Our experiments show that for 32-copy data replication, BT-Duper achieves a significant 62-76% reduction in latency compared to chain replication, while achieving a $60-80 \times$ performance improvement over OpenMPI’s Bcast in high load CPU environments.