Distributed Deep Learning of ResNet50 and VGG16 with Pipeline Parallelism

Data parallel distributed deep learning has been used to accelerate the learning speed. The communication is becoming a bottleneck as the computation time is accelerated. Therefore, the communication time is needed to be reduced. As a new acceleration method, research activity for pipeline parallelism has increased since 2018. However, the existing methods generally combine data parallelism and pipeline parallelism, and there has been a little analysis of speed improvement in pipeline parallelism alone. This paper analyzes the effect of speed improvement by pipeline parallelism. We compared the execution time of data parallelism and pipeline parallelism with VGG16 and ResNet50, respectively. VGG16 includes more parameters comparing to ResNet50 when the size of input data is the same. A comparison of the execution time shows that data parallelism is 3.3 times faster than pipeline parallelism for ResNet50. Pipeline parallelism is 2.45 times faster than data parallelism for VGG16. The speed-up of pipeline parallelism was significant in a model with a large number of parameters. Therefore, pipeline parallelism is suitable for large-scale models that have a large number of parameters.