Effective Evaluation of Multi-core Based Systems

This work proposes a practical technique to reduce the evaluation cost of multi-core based systems, when these systems are evaluated with parallel benchmarks. The proposed technique highlights the amount of redundancy in a set of parallel benchmarks and reduces this set to a subset of benchmarks such that: (i) the selected benchmarks are representative or non-redundant - i.e., the series of performance attained by any couple of representative benchmarks on different systems significantly differ, (ii) system evaluation is executed efficiently - i.e., on the system under evaluation, the average performance of representative benchmarks closely approaches the average performance of the whole suite. The proposed technique is validated with the industry-standard benchmark suites SPEC OMP2001 and SPEC OMP2012 on the largest data set of systems publicly available on the SPEC website - until the last quarter of the year 2012. For each suite, the proposed technique (i) identifies a subset of representative- benchmarks and (ii) shows how this subset of representative benchmarks - ≈ 50% of the total number of benchmarks - can be deployed to evaluate multi-core based systems with a prediction errors <; 5% at 99% confidence level.