Improving Join Reordering for Large Scale Distributed Computing

SQL Join is one of the most commonly used operators for workloads running upon SQL Engines built for Big Data like Apache Spark SQL, Apache Hive and Presto. As Joins on Big Data can be expensive, Join optimizations can significantly improve the efficiency and performance of it's workloads. In this work, we would address the problem of finding optimal order for SQL Joins, which is a well researched NP-Hard problem for traditional workloads. Traditional Cost Based Optimization in Query processing, including Join Reordering algorithms are not effective on Big Data due to lack of statistics. Statistics are mostly absent in Big Data as they are expensive to compute and needs frequent recompute due to velocity of data change. Hence, ensuring optimal order of Joins for Big Data is still largely a manual process done via trial and error.We are proposing a novel greedy algorithm for Join Reordering which can work in the absence of statistics. Proposed technique is of linear complexity in terms of number of joins. We have observed that on TPC-DS benchmarks it can reorder 39 queries and improve query performance upto 51%.