Magpie: Efficient Big Data Query System Parameter Optimization based on Pre-selection and Search Pruning Approach

With the rapid development of big data technology, big data systems are more and more used to process SQL query jobs. However, intricate big data system parameters have big effects on the quality of service for big data query performance. Moreover, the types of query jobs and data sizes of data sets are diversified, which have different optimal parameters for best running performance. There exist high complexity and difficulties in tuning the big data system parameters for efficient query job execution. To address this issue, we propose Magpie, an efficient big data query tuning system based on the genetic algorithm with customized pruning strategies. The experimental results show that the proposed approaches can reduce big data query job execution time by 35.5% on average compared with the cuttingedge big data system parameter tuning methods.