Memory mapped networks: a new deal for distributed shared memories ? the SciFS experience

Distributed Shared Memories (DSM) performance has always suffered from high network latencies and software communication layers with a large overhead. Memory mapped networks such as Scalable Coherent Interface (SCI) allow to reliably access remote memory without involving the operating system. To show how DSM systems can benefit from this technology, we have developed SciFS, a DSM tightly integrated with the operating system, that exploits the high performance and the remote memory access capabilities of SCI. We first show the respective advantages of two communications techniques with SCI: programmed IO (PIO) and remote DMA (RDMA). Then, we describe how to build a scalable page transfer mechanism by mixing PIO and RDMA. Despite the lack of a broadcast mechanism with SCI, we demonstrate that it is possible to build scalable synchronization primitives using PIO. Finally, we evaluate various consistency models with scientific computing applications from the Splash benchmark. We observe that, even if the rough network performance is good, it is not sufficient to obtain acceptable results with applications that require fine grain parallelism. However, we show that memory mapped networks provide an efficient hardware support to implement software DSM systems without requiring complex relaxed consistency models. This way, DSM design can be greatly simplified using this technology.