SmartPipe: Intelligently Freezing Layers in Pipeline Parallelism for Distributed DNN Training

Deep Neural Network (DNN) models have been widely utilized in various applications. However, the growing complexity of DNNs has led to increased challenges and prolonged training durations. Despite the availability of high-performance computing systems, certain DNNs still require several days for successful training. This study aims to address this issue by proposing a method for significantly reducing the training time of deep learning models while maintaining test accuracy. Existing approaches primarily concentrate on optimizing training efficiency through computational and communication overlap/scheduling. In contrast, this research takes a step further by inspiring transfer learning. Transfer learning is a useful way to quickly retrain a model on new data without having to retrain the entire network. During transfer learning, the first layers of the network are frozen while leaving the end layers open to modification. By doing so, computation and communication requirements in these frozen layers are eliminated. This intelligent approach involves freezing some of the specific DNN layers and allocating resources to the remaining active layers during the training process, thereby minimizing DNN training time. To achieve this objective, we propose an intelligently freezing DNN using pipeline parallelism. Through trace-based simulation results, our scheme has demonstrated its effectiveness in efficiently reducing the time cost of a training iteration.