Sparrow Tracer: Scalable Real Time Metrics from Event Log Pipelines at Twitter

Streaming event pipelines are one of the core components of Twitter's Data Infrastructure [1]. Twitter Sparrow is a project responsible for aggregating, processing and delivering user action generated events from microservices to data warehouses and data lakes in real time. User action generated events are converted into datasets used for data processing and data analytics use cases. This project is built using different on premise and cloud services. One of the important requirements of such a streaming event pipeline is the ability to measure important metrics such as latency, event count, event drop vs. success rate, and more. These metrics are responsible for defining the health of the streaming pipeline as well as providing valuable insights to users of the events. In this paper we introduce Sparrow Tracer which is a novel way to capture the event metrics using the concept of tracer events.