Study and Implementation of Elastic Stream Computing In The Cloud

In recent years, as the development of SNS, data is produced more and more quickly. "Store-and-process", the old style of data processing is not suitable now. Obviously, stream processing systems are more capable of such tasks. But currently, these systems only run on the fixed resource, they cannot take advantage of the "pay as you go" feature which is provided by the cloud. Besides, you have to adjust the parallelism in advance to survive the input data rate change. In this paper, we present a novel approach to combine the advantages of stream processing systems and cloud computing. Our techniques focus on handling dynamic streams whose volume are not predictable by running stream processing systems in the cloud. In our approach, we continuously monitor the workload of every tasks and each worker machine, adjust the parallelism of overload tasks, and the scale of the cluster automatically. In addition, we also propose and implement an efficient algorithm to keep load balance between machines.