Toward Reliable and Rapid Elasticity for Streaming Dataflows on Clouds

The pervasive availability of streaming data is driving Fast Data platforms for low-latency streaming applications. Such applications need to respond to dynamism in the input rates and task behavior using scale-in and -out on elastic Cloud resources. Platforms like Apache Storm do not provide robust means to respond to such dynamism and for rapid task migration across VMs. We propose several dataflow checkpoint and migration approaches that allow a running streaming dataflow to migrate, without any loss of in-flight messages or their internal tasks states, while reducing the time to recover and stabilize. We implement these strategies on Storm and evaluate them using micro and application dataflows for scaling in and out on 2 - 21 Cloud VMs. Our results show that we can migrate large dataflows and catchup with their processing 75% faster than Storm, which takes over 140 secs. We also find that our approaches stabilize the application up to 42% faster, and there is no failure and re-processing of messages.