CANDY: Enabling coherent DRAM caches for multi-node systems

This paper investigates the use of DRAM caches for multi-node systems. Current systems architect the DRAM cache as Memory-Side Cache (MSC), restricting the DRAM cache to cache only the local data, and relying on only the small on-die caches for the remote data. As MSC keeps only the local data, it is implicitly coherent and obviates the need of any coherence support. Unfortunately, as accessing the data in the remote node incurs a significant inter-node network latency, MSC suffers from such latency overhead on every on-die cache miss to the remote data. A desirable alternative is to allow the DRAM cache to cache both the local and the remote data. However, as data blocks can be cached in multiple DRAM caches, this design requires coherence support for DRAM caches to ensure correctness, and is termed Coherent DRAM Cache (CDC). We identify two key challenges in architecting giga-scale CDC. First, the coherence directory can be as large as few tens of MB. Second, cache misses to the read-write shared data in CDC cause longer delay due to the need to access the DRAM cache. To address both problems, this paper proposes CANDY, a low-cost and scalable solution that consists of two techniques for these challenges. First, CANDY places the coherence directory in 3D DRAM to avoid SRAM storage overhead, and re-purposes the existing on-die coherence directory as a DRAM-cache Coherence Buffer to cache recently accessed directory entries. Second, we propose Sharing-Aware Bypass, which dynamically detects the read-write shared data at run-time and enforces such data to bypass the DRAM cache. Our experiment on a 4-node system with 1GB DRAM cache per node shows that CANDY outperforms MSC by 25%, while incurring a negligible overhead of 8KB per node. CANDY is within 5% of an impractical system that has a 64MB SRAM directory per node, and zero cache latency to access the read-write shared data.