A Distributed Training Quantization Strategy for Low-speed and Unstable Network

Large-scale distributed computing environments are becoming the basis for training large-scale language models (LLMs). While network bandwidth is often considered a bottleneck for distributed training. The focus of gradient compression is on training LLMs using common computing resources (often with low bandwidth and unstable networks), especially as the demand for democratized AI continues to grow. In this case, the cost of training time increases disproportionately when the quality of the network degrades. To strike a balance between model performance and total training time, we propose the automatic quantization strategy. Considering factors such as bandwidth, training loss, and training time, the strategy dynamically adjusts the quantization throughout the training process. In our experiments, our method significantly reduces the total training time compared to the benchmark method.