TLPipe: An Efficient Two-Level Optimization Pipeline Model Parallelism for Large Model Training

Pipeline model parallelism has made large-scale model training feasible, but traditional pipeline parallel methods still suffer from significant bubble time during training as model parameters grow exponentially. This paper introduces the innovative pipeline method TLPipe, which employs a two-level optimization approach consisting of pipeline depth decoupling and on-demand partitioning to significantly reduce bubble time and enhance pipeline parallel efficiency. Additionally, a theoretical execution time model for pipeline performance evaluation and an automatic partitioning algorithm based on dual-chromosome genetic algorithm are designed. Both methods are combined to enable TLPipe to find the optimal partitioning strategy from a vast solution space. The performance of TLPipe is evaluated on a server equipped with 8 GPUs using the BERT series models and the IMDB dataset. Experimental results demonstrate that TLPipe excels in improving throughput, achieving throughput improvements of up to 56.18%, 57.93%, and 22.53% compared to GPipe, DAPPLE, and Chimera, respectively.