Balancing Accuracy and Evaluation Overhead in Simulation Point Selection

Simulators are the tool of choice when designing new computer architectures. While software-based simulators are (relatively) easy to adapt to evaluate new architectural mechanisms and can provide extensive non-invasive performance measurements, they are slow, and executing benchmarks to completion with realistic input sets is practically impossible. FPGA-accelerated simulators are orders of magnitude faster but require hardware implementations of all simulated components and provide limited performance measurements.In this paper, we first propose TraceDoctor to enable extensive non-invasive performance measurement within the FPGA-accelerated FireSim simulator. TraceDoctor enables tracing any architectural signal and keeps overheads in check by enabling architects to implement analysis-specific workers which run on the host computer to filter or compress trace data. We then use TraceDoctor to better understand which part(s) of a benchmark to focus on in software-based simulators, i.e., explaining how architects should configure the SimPoint methodology to yield simulation points that accurately predict whole-benchmark performance while minimizing the number of simulated instructions. Our analysis yields a number of interesting insights. For example, we find that evaluating relatively many, relatively small simulation points accurately predicts whole-benchmark performance while requiring to simulate relatively few instructions. In contrast, the current typical practice of selecting a single large simulation point yields low accuracy yet requires simulating many instructions.