ZipLine: An Optimized Algorithm for the Elastic Bulk Synchronous Parallel Model

The bulk synchronous parallel (BSP) is a celebrated synchronization model for distributed training of deep learning models. A shortcoming of the BSP is that it requires workers to wait for the straggler at every iteration. Therefore, employing BSP increases the waiting time of the faster workers of a cluster and results in an overall prolonged training time. To ameliorate this shortcoming of BSP, we proposed ElasticBSP [1], a model that aims to relax its strict synchronization requirement with an elastic synchronization by allowing delayed synchronization to minimize the waiting time. ELASTICBSP is realized by the algorithm named ZipLine. In this work, we show the theoretical proof of ZipLine and further propose algorithmic and implementation optimizations of ZipLine, namely ZipLineOpt and Ziplineoptbs, which reduce the time complexity of ZipLine to linearithmic time. The experiments show that ZipLineOpt and ZipLineOptBs enable the scalability of ElasticBSP. Further experimental evaluation on large deep neural networks on large ImageNet dataset demonstrate that our proposed Elas-ticbspmodel, materialized by the proposed optimized ZipLine variants, converges faster and to a higher accuracy than the predominant BSP.