DiasDNN-VD: Divide-and-Conquer Model for Asynchronous Training of Large-Scale DNNs Using Variational Dropouts

Machine learning and deep learning have taken significant strides, finding applications in various domains such as physics, biology, finance, agriculture, and many others. As models grow in complexity, training can take days, necessitating parallel frameworks. However, existing parallelism methods face GPU memory limits, large communication frequency and volume, and synchronization bottlenecks, restricting scalability. In this work, we propose a divide-and-conquer model for asynchronous DNN training across multiple devices. By forming small models from a given large model using variational dropout, DiasDNN-VD enables efficient execution. Experiments on multiple GPUs with various models and datasets show DiasDNN-VD outperforms PyTorch's data parallelism, achieving higher accuracy and up to 2x speedup.