An Innovative Communication Method for Distributed Large Model Training Based on Compression and Parallel Communication

With the rapid development of artificial intelligence technology, the need of large model training is becoming more and more urgent. However, the traditional large model training method has high communication cost, resulting in slow training speed and low efficiency. To solve this problem, this paper proposes a distributed large model training efficient communication scheme (DCoPa-DL) based on data compression and parallel communication. By compressing the training data and model gradient, and adopting parallel communication technology, the scheme effectively reduces the communication overhead and significantly improves the training speed. By verifying on the science and technology innovation infrastructure, the results show that the DCoPa-DL scheme can shorten the training time about 50%, compared with the original scheme, increase the training throughput by about 1 times, and slightly improve the model accuracy.