Enhancing Training Efficiency: A Novel Approach to Handling GPU Failures in Large-Scale Distributed System for LLM Training

In large-scale distributed machine learning systems for LLM Training, GPU failures can significantly disrupt training processes, often necessitating restoration from prior checkpoints. This paper proposes an alternative approach to handle GPU failures by ignoring updates from the batches processed by the failed GPUs and rescheduling these batches at a later feasible time. The rescheduling can occur at a subsequent time step, the final time step, or as part of a prologue to pre-training. This method aims to minimize the downtime and computational overhead associated with checkpoint restoration, thereby enhancing the efficiency and robustness of the training process. Experimental results demonstrate that this approach maintains model accuracy while reducing the time and resources required for recovery from GPU failures.