RAPID: A Rapid Automatic Parallelizer for Immense Deep Neural Networks

Deep Neural Network (DNN) frameworks need parallelism plans to execute immense models. The computed plans often combine data, model, and pipeline parallelism. Unfortunately, due to the intractable property of the problem, the current parallelism planners often fail to derive plans for immense DNNs. They either rely on experts to generate plans manually or profiling for their evaluation, making planners expensive and sub-optimal. We propose RAPID, an automatic parallelism planner for immense DNNs driven by a hierarchical abstract machine model. This model enables the design of a symbolic-based cost model that achieves robust prediction of parallelism cost with symbolic simplification. RAPID divides the parallelization problem hierarchically and symmetrically into linear-time sub-problems. We prove that the composition of the sub-problem solutions is optimal. Large-scale cluster experiments show that RAPID can reduce the planning time of immense DNNs (e.g., BERT) by up to 67× compared to state-of-the-art planners; while exhibiting high performance that matches expert-optimized plans.