PETS: Bottleneck-Aware Spark Tuning with Parameter Ensembles

Spark tuning with its dozens of parameters for performance improvement is both a challenge and time consuming effort. Current techniques rely on trial-and-error or best guess utilizing expert knowledge that very few posses. Previous tuning works are not compatible with Spark and also ignore the underlying problem of resource bottlenecks that is both the cause of performance issues, and a potential ally, if its awareness is leveraged in directing tuning to be more effective. We propose and develop PETS, a new method that allows the tuning of associated parameters at the same time, using resource bottleneck awareness to adjust parameter ensemble values in few iterations. Performance evaluation based on testbed implementation shows that with the use of PETS, representative workloads achieve: (1) Significant speedups; (2) Fast convergence speed; (3) Performance gains that are stable with varying workload data sizes, homogenous and heterogenous clusters, and initial parameter settings. The results show that PETS outperforms a machine learning based method, and achieves speedups of up to x4.78 and convergence speed as low as 2 iterations.