Model-Based Scheduling for Stream Processing Systems

Stream processing is emerging to react to the changing business situations of real-time processing. The main aim of this paradigm is to deal with the huge volume of data in the format of information flows originating from distributed devices. This consequently poses challenges to the scheduling problem in cloud data centers regarding the time-varying velocity of data ingesting and processing. In response to the uncertainties and complexities of streaming data, we propose a model-based scheduling scheme for stream processing systems, capturing the system behavior and providing an optimal allocation strategy to adapt to the changing work conditions. The proposed scheduling policy is implemented in Apache Storm, and micro-benchmarks with various shapes (e.g line, star, and diamond) were used in the evaluation. A topology that tracks trending topics on Twitter is also used, where the input is feeding with tweets in real-time. Experimental results show that the proposed solution can perform estimations that are well aligned with the system performance. The proposed scheduling policy achieves an improved performance with regards throughput and latency under varying ingesting rates.