Addressing workload variability in architectural simulations

The inherent variability of multithreaded commercial workloads can lead to incorrect results in architectural simulation studies. Although most architectural simulation studies ignore space variability's effects, our results demonstrate that space variability has serious implications for architectural simulation studies using multithreaded workloads. The standard solution - running long enough - does not easily apply to simulation because of its enormous slowdown. To address this problem, we propose a simulation methodology combining multiple simulations with standard statistical techniques, such as confidence intervals and hypothesis testing. This methodology greatly decreases the probability of drawing incorrect conclusions, and permits reasonable simulation times given sufficient simulation hosts.