ADT: An Additive Delta-Tuning approach for parameter-efficient tuning in pre-trained language models

The advent of pre-trained language models (PLMs) has revolutionized the field of natural language processing (NLP), enabling models to leverage vast amounts of language and world knowledge for various downstream tasks. However, the conventional fine-tuning of these models requires updating full parameters, which is computationally intensive and impractical in resource-constrained settings. Parameter-efficient tuning methods have been developed to mitigate this. However, these methods may need to be improved in terms of inference latency, optimization difficulty, and input sequence length reduction.In this paper, we introduce a new reparameterization-based parameter-efficient tuning method, Additive Delta Tuning (ADT). ADT only fine-tunes part of the row and column parameters in the PLM weight matrix. This approach significantly reduces the computational cost compared to existing methods, such as Low-Rank Adaptation (LoRA), while maintaining competitive performance. We conducted extensive experiments across multiple natural language understanding (NLU) tasks, including text classification, text entailment, named entity recognition, etc., to evaluate the effectiveness of ADT. Our experimental results demonstrate that ADT achieves comparable or better performance than traditional fine-tuning methods, requiring less than 0.3% parameter updates. This shows that ADT is an effective method for parameter-efficient tuning. Our method achieves comparable results to LoRA on multiple NLU tasks, with the added benefit of requiring less computational cost.