TranLogs: Lossless Failure Recovery Empowered by Training Logs

When running deep learning training jobs, in order to prevent training loss due to softwarelhardware failures, a checkpointing mechanism is usually used to periodically store snapshots of the training model in non-volatile memory to facilitate recovery from failures. However, the recovery effect of this method is closely related to the checkpointing period and the failure time, and in the worst case, even nearly one period of training accuracy will be lost. With increasing model sizes, the need to develop more fine-grained model fault recovery methods to reduce model fault loss is more urgent for deep learning training tasks that use checkpointing mechanisms as fault recovery guarantees. We present TranLogs, a checkpoint failure recovery method combined with training logs for lossless failure recovery when training deep learning models. TranLogs proposes to use logging to supplement the checkpoint file and track the iterative process of model training parameters. For the situation where the checkpoint file is not updated in time and encounters model failures, resulting in a large loss during recovery, TranLogs can handle it effectively. The best-case scenario achieves about 24 % and 6% improvement in the loss and accuracy of the recovered training state, respectively, which is closer to the state already achieved by the model training at the moment of failure.