Dynamic load distribution in the Borealis stream processor
Xing, Y.; Zdonik, S.; Hwang, J.-H.
Data Engineering, 2005. ICDE 2005. Proceedings. 21st International Conference on
Volume , Issue , 5-8 April 2005 Page(s): 791 - 802
Digital Object Identifier 10.1109/ICDE.2005.53
Summary: Distributed and parallel computing environments are becoming cheap and commonplace. The availability of large numbers of CPU's makes it possible to process more data at higher speeds. Stream-processing systems are also becoming more important, as broad classes of applications require results in real-time. Since load can vary in unpredictable ways, exploiting the abundant processor cycles requires effective dynamic load distribution techniques. Although load distribution has been extensively studied for the traditional pull-based systems, it has not yet been fully studied in the context of push-based continuous query processing. In this paper, we present a correlation based load distribution algorithm that aims at avoiding overload and minimizing end-to-end latency by minimizing load variance and maximizing load correlation. While finding the optimal solution for such a problem is NP-hard, our greedy algorithm can find reasonable solutions in polynomial time. We present both a global algorithm for initial load distribution and a pair-wise algorithm for dynamic load migration.
View citation and abstract |
Cart
