Move Real-Time Data Analytics to the Cloud: A Case Study on Heron to Dataflow Migration

Twitter is migrating its real-time data analytics infrastructure to the cloud. We propose a complete cloud migration procedure to tackle five identified challenges. Following the proposed solution, we successfully migrated a typical Heron job to Dataflow recently. This use case implements 1) control plane of provisioning and orchestration, 2) data plane of Heron to Beam programming model translation, 3) job IOs through replication and direct access/proxying, and 4) data validation by control test and metrics monitoring. The evaluation demonstrated the new Dataflow job matches the legacy Heron job in terms of performance and SLAs. Finally, we retrospect the migration process and identify future work on automated migration.