Popper: A Dataflow System for In-Flight Error Handling in Machine Learning Workflows

We present Popper, a dataflow system for building Machine Learning (ML) workflows. A novel aspect of Popper is its built-in support for inflight error handling, which is crucial in developing effective ML workflows. Popper provides a convenient API that allows users to create and execute complex work-flows comprising traditional data processing operations (such as map, filter, and join) and user-defined error handlers. The latter enables inflight detection and correction of errors introduced by ML models in the workflows. Inside Popper, we model the workflow as a reactive dataflow, a directed cyclic graph, to achieve efficient execution through pipeline parallelization. We demonstrate the inflight error-handling capabilities of Popper, for which we have built a graphical interface, allowing users to specify workflows, visualize and interact with its reactive dataflow, and delve into the internals of Popper.