CURRF: A Code-Based Framework for Faithful Replay Distributed Applications

Debugging distributed system is programmer's nightmare be-cause of non-determinism bugs. Those non-repeatable bugs force developers back to the outdated and time consuming techniques such as printf and log mining for investigation. To relieve this issue, record and replay mechanisms have been proposed. These methods allow developers use their cyclic debug skill in non-deterministic situations. In this pa-per, we present the design and implementation of CURRF: a code-based, fully user-space light-weighted record and replay framework. In CURRF, developers can easily re-play individual processes in a large-scaled distributed system without touching other components. It achieves this goal by introducing a code annotation mechanism. Programmers can write the annotations in the source code and notify the logger for the critical non-deterministic operations. CURRF is much more flexible and easier to use than previous record and replay solutions such as R2. In this paper, we demonstrate the efficiency and usefulness of CURRF for new applications as well as legacy programs. The experiment results show that CURRF introduces very little interference when the program is running in debugging mode.