Impact of Resource Millibottlenecks on Large-Scale Time Fluctuations in Spark SQL

Simultaneously, it is always an important goal of Spark SQL to execute SQL query statements quickly, efficiently and stably. Although many studies have focused on improving the performance of Spark SQL, few are devoted to improving the impact of resource bottlenecks on performance. To address this problem, this paper makes an in-depth study on the performance impact of Spark SQL execution for resource bottlenecks, especially millibottleneck. First, based on the TPCDS benchmark, this paper conducts extensive experimental tests of Spark SQL and finds that during the period of high resource utilization, the response time of Spark SQL applications often fluctuates on a large scale (for example, from tens of milliseconds to a few seconds). Following the steps of Elba, we leverage the millibottlneck theory of performance bugs to make the connections between the resource millibottleneck and the performance of Spark SQL. Specifically, based on the understanding of "Spark execution mechanism", and looking inside the Spark applications, this paper analyzes, locates and extracts key processes of Spark SQL statements. Furthermore, start and finish timestamps of abnormal tasks are extracted, and the utilization of various type of resources are calculated over these periods. The millibottleneck theory is applied to make connections between the key processes of execution logic of abnormal tasks and these utilization in fine granularity. Finally, the execution logic is analyzed and revealing the root causes of Spark SQL performance bottlenecks. Through extensive experiments, we find that when the utilization of CPU, the quantity of memory page events and memory usage increased sharply in a very short time, the response time of the application shows a positive correlation fluctuation. Applying the millibottleneck theory of performance bugs, we may make conclusion that the execution performance of Spark will be seriously affected when there are millibottlenecks in CPU, large increment in page events and page usage in memory.