R-Max: A Method for Approximating the Benefit of Ideal Prefetching and Replacement Policy

Memory performance continues lagging behind the demand of processing elements, a well known phenomenon known as the memory wall. Cache prefetching is a well studied and effective method to bridge this gap. Despite a long history of study and the existence of many prefetchers, an open question remains with respect to the upper bound of performance that might be had from prefetching. Here, we propose a system, R-Max, to approximate ideal prefetching and replacement policy with realistic constraints on bandwidth, cache structure, and capacity but oracular knowledge of future accesses. We compare R-Max’s approximated ideal speedup against the speedup of current state of the art prefetchers to show how much remaining performance gain may be left on the table for cache prefetching techniques. We show that, for a set of workloads taken from SPEC CPU2017, CVP, GAP and XSBench, up to 299.6% maximum and 72.6% average gains are possible under realistic assumptions for a prefetcher that perfectly predicts future accesses, outperforming current state of the art prefetchers by 60.8%. Interestingly, we see that the workloads where R-Max shows the most potential have little relationship with those where existing prefetchers perform best. Taken together our results highlight the need for new research into prefetching techniques for these under-exploited workloads.