By Topic

An effective algorithm for parallelizing hash joins in the presence of data skew

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$31 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

3 Author(s)
Wolf, J.L. ; IBM Thomas J. Watson Res. Center, Yorktown Heights, NY ; Dias, D.M. ; Yu, P.S.

A parallel hash join algorithm based on the concept of hierarchical hashing is proposed to address the problem data skew. The proposed algorithm adds an extra scheduling phase to the usual hash and join phases. During the scheduling phase, a heuristic optimization algorithm, using the output of the hash phase, attempts to balance the load across the multiple processors in the subsequent join phase. The algorithm naturally identifies the hash partitions with the largest skew elements, splits them up, and assigns each of them to an optimal number of processors. Assuming a Zipf-like distribution of the elements in the join column, the algorithm is shown to achieve good load balancing for the join phase in a CPU-bound environment, and it is shown to be fairly robust relative to the degree of data skew and the total number of processors. The overall speedup due to this algorithm is compared with conventional parallel hash join methods and the considerable advantage of the proposed method is demonstrated even when the additional scheduling phase is taken into account

Published in:

Data Engineering, 1991. Proceedings. Seventh International Conference on

Date of Conference:

8-12 Apr 1991