Keyed Watermarks: A Fine-grained Tracking of Event-time in Apache Flink

Big Data Stream processing engines such as Apache Flink use windowing techniques to handle unbounded streams of events. Gathering all pertinent input within a window is crucial for event-time windowing since it affects how accurate results are. A significant part of this process is played by watermarks, which are unique timestamps that show the passage of events in time. However, the current watermark generation method in Apache Flink, which works at the level of the input stream, tends to favor faster sub-streams, resulting in dropped events from slower sub-streams. In our analysis, we found that Apache Flink’s vanilla watermark generation approach caused around 33% loss of data if 50% of the keys around the median are delayed. Furthermore, the loss surpassed 37% when 50% of random keys are delayed.In this paper, we present a novel strategy called keyed watermarks to overcome data loss and increase the accuracy of data processing to at least 99% in most cases. We enable separate progress tracking by creating a unique watermark for each logical sub-stream (key). In our study, we outline the architectural and API changes necessary to implement keyed watermarks and discuss our experience in extending Apache Flink’s enormous code base. Additionally, we compare the effectiveness of our strategy against the conventional watermark generation method in terms of the accuracy of event-time tracking.

I. Introduction

The proliferation of IoT-connected devices surpassing 70 billion presents innumerable potential and significant problems for data stream processing and computing infrastructure [1] We have seen the rise of numerous distributed stream processing systems, a.k.a. stream processing engines (SPE) over the past few years since processing massive amounts of data at fast speeds has grown more difficult. Examples of such systems are Apache Spark Streaming [2], Apache Flink [3], Apache Samza [4], to name just a few. Such systems provide scalable (distributed) processing for data streams that are fault-tolerant and in real-time fashion according to the time at which the data was generated.