Evaluating Machine Learning Models for Disparate Computer Systems Performance Prediction

Performance prediction is an active area of research due to its applicability in the advancements of hardware-software co-development. Several empirical machine-learning models, such as linear models, non-linear models, probabilistic models, tree-based models and, neural networks, are used for performance prediction. Furthermore, the prediction model's accuracy may vary depending on performance data gathered for different software types (compute-bound, memory-bound) and different hardware (simulation-based or physical systems). We have examined fourteen machine-learning models on simulation-based hardware and physical systems by executing several benchmark programs with different computation and data access patterns. Our results show that the tree-based machine-learning models outperform all other models with median absolute percentage error (MedAPE) of less than 5% followed by bagging and boosting models that help to improve weak learners. We have also observed that prediction accuracy is higher on simulation-based hardware due to its deterministic nature as compared to physical systems. Moreover, in physical systems, the prediction accuracy of memory-bound algorithms is higher as compared to compute-bound algorithms due to manufacturer variability in processors.