Application-Level Determinism in Distributed Systems

Deterministic and reproducible program execution eases the development and debugging of distributed systems. However, deterministic execution comes at high performance costs and is hard to achieve, especially when running on different hardware. In this paper we introduce the concept of application-level determinism and describe how the parallel programming model Spawn & Merge can be used for scalable and deterministic distributed computation. Application-level deterministic applications yield reproducible deterministic results independent of the number of nodes participating in the computation, even though intermediate tasks may be executed in an unpredictable schedule. To achieve consistency independent of the order in which operations have been applied we present a new Operational Transformation algorithm, which mitigates the performance loss of introducing determinism with Spawn & Merge. We show that such deterministic processing can scale across a cluster of compute nodes and discuss for which kind of workload the programming model is feasible. Furthermore, for high and low workloads, we evaluate the cost of adding determinism to be 28% and 40% higher than perfect parallel computation.