PanDa: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation

Prompt Transfer (PoT) is a recently-proposed approach to improve prompt-tuning, by initializing the target prompt with the existing prompt trained on similar source tasks. However, such a vanilla PoT approach usually achieves sub-optimal performance, as (i) the PoT is sensitive to the similarity of source-target pair and (ii) directly fine-tuning the prompt initialized with source prompt on target task might lead to forgetting of the useful general knowledge learned from source task. To tackle these issues, we propose a new metric to accurately predict the prompt transferability (regarding (i)), and a novel PoT approach (namely
PanDa
) that leverages the knowledge distillation technique to alleviate the knowledge forgetting effectively (regarding (ii)). Extensive and systematic experiments on 189 combinations of 21 source and 9 target datasets across 5 scales of PLMs demonstrate that: 1)
our proposed metric works well to predict the prompt transferability
; 2)
our

PanDa
consistently outperforms the vanilla PoT approach by 2.3% average score (up to 24.1%) among all tasks and model sizes
; 3)
with our

PanDa
approach, prompt-tuning can achieve competitive and even better performance than model-tuning in various PLM scales scenarios
.