Recommendation engine for optimal Spark delta tables partitioning

The efficiency of a Spark-based ETL system relies significantly on the optimal partitioning of numerous Delta Tables, taking into account the query patterns of end-users and dataset characteristics. These factors are dynamic and challenging to predict in advance, emphasizing the need for an automated recommendation system for the current optimal partitioning of Delta Tables. Our work addresses this by developing a recommendation system that analyzes Spark logs and suggests the optimal partitioning and implementation of Z-ordering. Our system employs various original recommendation models, with each model contributing votes on dataset columns for partitioning and the applicability of Z-ordering. The recommendation system selects the optimization that carries more votes. To evaluate the efficiency of our system, we conducted benchmarks against another Spark recommendation engine for column partitioning developed by Gou et al. The results demonstrate that our system achieves almost identical performance compared to theirs. Our system provided a speed up between [17%, 19%] compared without the use of Z-order or explicitly partitioning, while their system provided an improvement between [19%, 20%]. However, our system has the great advantage of leveraging the readily available Spark logs integrated into the Spark platform, while Gou’s approach necessitates collecting and analyzing inputs that are relatively complex.