SparkPerf: A Benchmarking Framework for Evaluating the Performance of Spark Data Analytics Projects

Machine learning is quickly becoming an integral part of modern business. Significant effort goes into developing and deploying machine learning models for estimation, economic forecasting, and client interaction. This effort includes planning for tools and platforms to be used in training and validating these models, as well as allocating resources, time, and budget for these tasks. However, this planning remains largely dependent on human acumen and is expensive to determine in a systematic fashion with automated tools. Benchmarking is the process of efficiently running experiments to determine a system's performance requirements, among others, in order to aid planning and resource allocation. Benchmarking intelligent and data-intensive systems remains in its infancy and does not cover fully realistic or particular case studies. Furthermore, existing approaches often focus on one-time batch evaluations, which may be inadequate for workloads requiring continuous processing over extended periods. In this work, we propose SparkPerf, a benchmarking tool specifically designed for machine learning applications deployed with Apache Spark, supporting both batch and transactional workloads. Unlike existing solutions, SparkPerf incorporates automated workload generation with synthetic data, providing a more comprehensive evaluation. Additionally, SparkPerf focuses on longitudinal transactional workloads, which represent a more realistic class of case studies for enterprises, and it offers high customizability, allowing users to test their own applications with their own, synthetically augmented, datasets. Our experiments demonstrate the benchmark's reliability, consistency, portability, and customizability.