From Static to Dynamic: A Deeper, Faster, and Adaptive Language Modeling Approach

Transformer-empowered Pre-trained language models (PrLM) have achieved great success in sumless natural language processing tasks. Meanwhile, keeping seeking better performance leads to deeper and deeper models, which is surely suboptimal by taking the same deep model for various input samples with different learnable complexities, let alone the unnecessity of too deep model for ‘easy’ input. Therefore in this work, we explore a dynamic use of Transformer layers. In detail, we propose a novel Transformer-the-Adaptive including an estimator module that enables the model to automatically learn and evaluate the complexity of every input samples, so that the model can adaptively use the most economic numbers of layers for inference. The proposed Transformer is implemented in the current state-of-the-art PrLM, ALBERT, giving a deeper, faster and adaptive model, ALBERTa. We train four-size optimized ALBERTa models including base, large, xlarge, and xxlarge and carry out experiments on three typical tasks MRC, NLU, and NER. Experiment results show that our proposed ALBERTa model optimized by Transformer-the-Adaptive has better performance and efficiency than the original ALBERT model, which fully reflects the effectiveness of Transformer-the-Adaptive.