Marlin: A memory-based rack area network

Disaggregation of hardware resources that are traditionally embedded within individual servers into separate resource pools is an emerging architectural trend in hyperscale data center design, as exemplified by Facebook's disaggregated rack architecture. This paper presents the design, implementation and evaluation of a PCIe-based rack area network system called Marlin, which is designed to support the communications and resource sharing needs of disaggregated racks. By virtue of being based on PCIe, Marlin presents a memory-based addressing model for both I/O device sharing among multiple hosts and inter-host communications. That is, when a node communicates with other nodes or accesses resources in the same rack, it uses memory read and write operations. In the area of inter-node communications, Marlin offers hardware-based remote direct memory access (HRDMA) as a first-class communications primitive between servers within a rack. In addition, Marlin supports socket-based communications for legacy network applications and cross-machine zero memory copying for applications designed specifically to take full advantage of memory-based communications. Empirical measurements on a fully operational Mar-lin prototype based on 4-lane Gen3 PCIe technology show that the one-way kernel-to-kernel latency is 8.5μsec and the end-to-end sustainable TCP throughput is 19.6 Gbps.