Single-Node Partitioned-Memory for Huge Graph Analytics: Cost and Performance Trade-Offs

Because of cost, non-volatile memory NVDIMMs such as Intel Optane are attractive in single-node big-memory systems. We evaluate performance and cost trade-offs when using Optane as volatile memory for huge-graph analytics. We study two scalable graph applications with different work locality, access patterns, and parallelism. We evaluate single and partitioned address spaces-Memory and AppDirect modes-and compare with distributed executions on GPU-accelerated and CPU-based supercomputers. We show that AppDIRECT can perform and scale better than Memory for the largest working sets (12%), even when dominated by irregular access patterns, if most accesses are NUMA-local and Optane accesses are frequently reads. Surprisingly, between Memory and AppDirect, processor-cache performance can change due to line invalidations; updates to the caching policy (via non-temporal hints) can make a 25% improvement. We observe that single-node graph analytics frequently has >4-10× cost/performance advantages over distributed-memory executions on supercomputers.