Selective Error Correction for Activation in Large Language Model Training

In Large Language Model (LLM) training, acti-vations constitute a significant portion of memory usage, and memory-side errors occurring in activations can lead to training failures. In this paper, we propose a lightweight but effective error correction method to mitigate the impact of such errors. The proposed approach leverages the inherent error tolerance of activations by selectively protecting only a subset of significant bits. Specifically, based on bit-flip error analysis, six sensitive bits are identified, and error-correcting codes (ECC) are applied exclusively to these bits. Compared to ECC implemented in HBM2, the proposed method exhibits 5.4 times lower parity overhead and 4 times higher area efficiency, while maintaining training performance in error-prone environments.