Accelerating Distributed Repartition Joins on Skewed Datasets via Patch-Based Shuffling

In distributed workloads involving joins and aggregations, skewed attribute values often cause load balancing issues, leading to stragglers and increased execution times. Existing solutions often rely on cost-based models, require extensive parameter tuning, or necessitate modifications to distributed execution engines, limiting their usability and generality. To address this challenge, we present a novel patch-based repartitioning algorithm that eliminates load imbalances while minimizing network overhead. Our approach extends the subset-replicate technique by leveraging data distribution and location statistics to optimize data locality and reduce unnecessary data movement. Unlike traditional hash-based methods, our technique is skew-insensitive, requires no parameter tuning, and integrates seamlessly into existing distributed execution engines as a drop-in replacement for the shuffle mechanism. The proposed method operates in three distinct stages: (1) statistics collection and load threshold computation, (2) patch-based subgroup assignment to ensure optimal load balancing with minimal replication, and (3) informed data shuffling and join execution. This structured process ensures even workload distribution across workers while reducing network I/O. Theoretical analysis proves the scalability, skew robustness, and load-balancing guarantees of our approach, establishing bounds on maximum worker load and network data movement. Experimental evaluations demonstrate that our method achieves perfectly balanced workloads and reduces execution time by up to 81% compared to conventional hash-based joins under moderate to high skew, while introducing negligible overhead at low skew levels. These improvements are attributed to reduced data movement, optimized worker utilization, and the algorithm’s robust theoretical foundation. Our research provides a versatile and practical solution for skewed data processing, significantly advancing the efficiency of distributed data management systems.