ME-BERT: Multi-exit BERT by use of Adapter

Multi-exit BERT is the backbone architecture for many inference speedup methods. However, its training procedure is not well studied. In this work, We propose a novel framework, Multi-exit BERT (ME-BERT), for improving the training procedure of multi exit BERT. First, through analysis of the two-stage training (2ST) procedure [1], we propose the Enhanced-2ST, which consists of two generic yet effective modifications to 2ST. Second, to further uncover the representation capabilities of the shallow layers in the second stage of Enhanced-2ST, we apply the multi-exit adapter to finetune the backbone layer-by-layer and provide more suitable representations to the intermediate exits. Extensive experiments are conducted on the GLUE benchmark, which shows that our ME-BERT can significantly outperform the state-of-the-art (SOTA) mssulti-exit BERT training methods.