Performance Prediction of Sparse Matrix Multiplication on a Distributed BigData Processing Environment

Sparse matrix multiplication (SPMM) is widely used for various machine learning algorithms. With advancements in big-data processing, the importance of distributed SPMM processing becomes important for handling large-scale datasets. We conducted thorough experiments using various distributed SPMM implementations and discovered considerable performance variations for distinct datasets and scenarios. To provide an optimal SPMM execution environment, we propose features that represent SPMM task characteristics. Using these features, we propose building a tree-based nonlinear gradient boosting (GB) regressor model that presents superb prediction accuracy across diverse distributed SPMM implementations and datasets.