HIVE+: An Enhanced High-Priority Victim Cache to Accelerate GPU Memory Accesses

The victim cache was originally designed as a small-capacity secondary cache to hold recently evicted lines from the L1 data cache in CPUs. However, this design is often suboptimal for GPUs due to the following two specific features of GPUs. First, a significant number of GPU applications have low hit rates in the L1 data cache, while the probability of lines evicted from the L1 data cache being subsequently reused is high. This means that the number of memory requests that hit in the victim cache can approach or even exceed the number of requests that hit in the L1 data cache. Second, GPUs use a unified architecture for the L1 data cache and shared memory, or the L1 cache and texture cache, to save hardware overhead. However, this hardware architecture significantly increases the L1 data cache access latency. We observe that some applications are susceptible to access latency, and increasing latency can severely degrade performance. To this end, we introduce HIVE+, an enhanced high-priority victim cache designed to accelerate GPU memory accesses. The key idea of HIVE+ is to exploit idle registers as the data array of the victim cache and place the tag array of the victim cache at a higher level in the memory hierarchy than the L1 data cache. Our experimental results show that HIVE+ achieves an average performance improvement of 47.5% and 35.0% compared to the baseline and Linebacker architectures, respectively.