The Case for Better Integrating Scalable Data Stores and Stream-Processing Systems

Scalable stream processing systems require external storage systems for long-term storage of non-emphemeral state. Recent research have pointed to scalable in-memory key-value stores, such as Redis, as an efficient solution to external management of state [1], [2]. While such data stores have been interconnected with scalable streaming systems, they are currently managed independently, missing opportunities for optimizations, such as exploiting locality between stream partitions and table shards, as well as coordinating elasticity actions.