NAMB: A Quick and Flexible Stream Processing Application Prototype Generator

The importance of Big Data is nowadays established, both in industry and research fields, especially stream processing for its capability to analyze continuous data streams and provide statistics in real-time. Several data stream processing (DSP) platforms exist like the Storm, Flink, Spark Streaming and Heron Apache projects, or industrial products such as Google MillWheel. Usually, each platform is tested and analyzed using either specifically crafted benchmarks or realistic applications. Unfortunately, these applications are only briefly described and their source code is generally not available. Hence, making quick evaluations often involves rewriting complete applications on different platforms. The lack of a generic prototype application also makes it difficult for a developer to quickly evaluate the impact of some design choices. To address these issues, we present NAMB (Not only A Micro-Benchmark), a generic application prototype generator for DSP platforms. Given a high-level description of a stream processing application and its workload, NAMB automatically generates the code for different platforms. It features a flexible architecture which makes it easy to support new platforms. We demonstrate the benefits of our proposal to quickly generate application prototypes as well as benchmarks used in published papers. Overall, our approach provides easily replicable, comparable and customizable prototypes for data stream platforms. Moreover, NAMB provides similar performance in terms of latency and throughput to existing benchmarks, while only requiring a simple high-level description.