PreAdapter: Sparse Adaptive Parameter-Efficient Transfer Learning for Language Models

As model parameters grow, fine-tuning new models for each task becomes resource-heavy. Parameter-Efficient Transfer Learning (PETL) offers a solution by updating minimal parameters, allowing large pretrained models to adapt to downstream tasks more efficiently. However, existing PETL methods often suffer from increased inference latency. This paper presents ‘PreAdapter’, an enhancement to traditional adapter structures, overcoming this latency issue. PreAdapter maintains PETL's benefits while avoiding extra inference delay. Specifically, we first demonstrate that adapter modules can be incorporated into large pre-trained language models through structural reparameterization. We explore the optimal deployment sites and the sparse architecture of these adapter modules, aiming to enhance the performance and parameter-efficiency of PreAdapter. To validate the effectiveness of PreAdapter, we perform transfer experiments on the GLUE dataset using the most representative BERT models. Compared to preceding adapter methods, PreAdapter updates 0.6 million fewer parameters in the 110 million parameter BERT-base model, yielding an average performance improvement of 2.7%. In the case of the 335 million parameter BERT-large model, PreAdapter reduces parameter updates by 1.5 million relative to previous methods, leading to an average performance enhancement of 1.7%.