Addition of CXL memory pool to increase LLM training check-pointing frequency

This paper presents a novel approach to enhance GPU cluster utilization and significantly reduce Large Language Model (LLM) training time by increasing checkpointing frequency. By introducing a Compute Express Link (CXL) based PC-DDR or LPDDR as an in-memory cache, we aim to reduce work loss during node-level failures, which occur more frequently than power loss or other issues. The proposed method utilizes a CXL DRAM pool for frequent checkpoints and a networked storage lake with enterprise SSDs for less frequent checkpoints (one-tenth). A CXL switch is employed to facilitate peer-to-peer (P2P) communication between CXL DRAM and SSD and an elastic memory model, making it suitable for various types of checkpointing techniques. The impact of model parallelization on read/write bandwidth and efficient GPU utilization at a node level is also discussed. While model parallelization can reduce bandwidth by 10 times, it also decreases effective GPU utilization due to smaller matrix computations. This work offers a practical solution to increase checkpointing frequency and reduce work loss during node-level failures, contributing significantly to LLM training efficiency.