Spark Parameter Optimization Based on HC2-SA Algorithm

Spark, as an efficient big data processing framework, the selection of its parameters is crucial for the performance and efficiency of the application. Inappropriate parameter configurations can seriously affect the performance of Spark jobs. This article proposes a Spark parameter optimization method based on the HC2-SA algorithm to improve the performance and efficiency of Spark applications. A series of Spark applications were experimented with different parameters, and the running time data was collected. Multiple machine learning algorithms were used to train and test the data to establish a performance model for Spark jobs. Then, the HC2-SA algorithm was used to search in the parameter space and obtain optimized parameters for Spark applications. Three different parameter space search algorithms, including hill climbing, simulated annealing, and HC2-SA algorithm, were compared in the experiment. The experimental results showed that the HC2-SA algorithm performed the best in Spark parameter optimization, and it could find the optimized parameter combination in a shorter time.