An Efficient Lossless Gradient Compression Technique for Distributed Training Systems

Distributed training improves the accuracy of large models by scaling parameter sizes. However, this scalability introduces communication bottlenecks between nodes. Gradient compression can alleviate these bottlenecks, but most existing methods are lossy, potentially compromising convergence and accuracy. This paper proposes a novel lossless compression algorithm for the floating-point gradient data, significantly improving compression efficiency compared to traditional lossless float compression algorithms while overcoming the shortcomings of lossy compression. First, new distance-run-length encoding and exponent prediction encoding using dynamic Huffman coding (D.Huff.) are designed based on the distinct statistical characteristics of zero and non-zero gradient data, effectively representing data values and zero locations. Second, an innovative hybrid Huffman coding (H.Huff.) is introduced, combining the advantages of dynamic and static Huffman coding. It enhances compression efficiency via local data adaptability, and simultaneously improves computational efficiency and hardware implementation friendliness with multiple fixed Huffman code tables. We also present a K-means-based probability vector clustering algorithm to effectively construct multiple prebuilt Huffman code tables. To evaluate the effectiveness of the proposed compression method, we selected two different deep neural network (DNN) learning tasks: (1) image classification, using ResNet50 and VGG16 for CIFAR-100 training, and ViT for ImageNet; and (2) language modeling, employing BERT on the IMDB dataset. The results are presented in Table 1. Furthermore, we tested mixed-precision training compression using the BF16 format. Experimental results demonstrate that our method achieves a lossless compression ratio of 0.4931 on typical gradient data, representing a significant 16.61% improvement over the state-of-the-art lossless float data compression approach, ndzip.