Towards Automatic Tuning of Apache Spark Configuration

Apache Spark provides a large number of configuration settings that may be tuned to improve the performance of specific applications running on the platform. However, it is non-trivial to identify the combination of settings that may improve the performance of a specific application as the influence of each setting on performance may vary across applications. As identifying the optimal combination of settings is computationally infeasible due to exponential search space, in this paper we investigate machine learning based approaches to construct application specific performance influence models, and use them to tune the performance of specific applications running on Apache Spark platform. We evaluated our approach using 9 different applications on a 6 node cluster and demonstrated that our framework can reduce execution time by 22.8% to 40.0% depending on applications.