An Analysis of Training Convolution Neural Networks for Image Classification Using Distributed Deep Learning

Deep learning algorithms have seen a growth in recent times due to their capability of analyzing hidden patterns in large datasets efficiently. However, an increase in the complexity of models, has made it challenging to train them on a single node, even if it is equipped with a Graphical Processing Unit(GPU). Distributed deep learning(DDL) has become an emerging branch of deep learning, where the joint compute resources of various machines connected together in a cluster are harnessed to train a single model. However, DDL faces certain limitations such as communication overhead and synchronization issues while merging gradients from various nodes. We propose a system to analyze the training performance of a deep learning model on a single node versus distributing on two nodes. To perform the analysis, data parallelism is implemented on three deep learning models using the MultiMirroredStrategy of Tensorflow. Results have shown that, while DDL underperforms with small sized models, it gives better speedup ratio and accuracy with larger, complex models.