Understanding Address Translation Scaling Behaviours Using Hardware Performance Counters

Virtual memory researchers are combining benchmark programs with synthetic input generators when crafting experiments. These generators are tuned to generate program instances with memory footprints in the gigabyte to terabyte range, with the intention that these instances will have considerable address translation overhead. Yet, the relationship between workload, input, and address translation overhead is poorly understood - complicating workload and input selection.We characterize this relationship on real machines across various workloads, input generators, and memory footprints using measured performance counter data. We observe that address translation overhead typically increases as the log of the memory footprint, but there are exceptions - even for workloads that are thought of as being "address translation intensive".We measure and provide results for the memory footprint scaling behavior of the program, translation lookaside buffers (TLBs), memory management unit (MMUs) caches, and page table walkers. We show that no individual factor is responsible for degrading performance but rather the combination of all four (and their interactions) gives rise to address translation-related slowdown. We find evidence for the existence of a TLB filtering effect: higher TLB hit rates can cause longer page table walks because the TLB filters page-level access patterns from the MMU caches and page table walker(s).We propose a new metric of address translation pressure called "walk cycles per instruction" that captures the effects of all the components and interactions. We demonstrate that it strongly reflects true address translation overhead.Finally, we show that misspeculated and aborted page table walks can constitute up to 57% of all initiated page table walks and that the problem gets worse with increasing memory footprint. Superpages can reduce the number of misspeculated and aborted walks, suggesting that superpages have effects beyond reducing TLB miss rate and shortening page table walks.