Improving Zero-Shot Generalization for CLIP with Prompt Ensemble self-Distillation

Prompt tuning has emerged as an effective alternative for adapting pre-trained Vision-Language Models (VLMs) to various downstream tasks. In our experiments utilizing prompt tuning methods, we observed that modifying the prompt initialization led to inconsistencies in the model’s predictions, particularly with pronounced variability on specific datasets. Motivated by this observation, we examine the predictive performance of two ensemble methods: prompt fusion and logits fusion. Experimental results indicate that logits fusion results in considerable performance improvements, while prompt fusion does not yield any enhancements. However, a significant downside of logits fusion is the enormous rise in inference time. To investigate a practical approach for integrating knowledge derived from multiple prompts without incurring additional inference costs, we propose a straightforward Prompt Ensemble self-Distillation (PED) framework that considerably improves the generalization capacity of prompt tuning. Specifically, we initialize multiple groups of prompts, and during the training process, we integrate the prediction outputs from each group to facilitate the learning of the fused prompts. The proposed self-distillation approach offers dual benefits: enhancing the performance of both the fused prompts and the fused logits. We utilize fused prompts for prediction during the inference process, thereby achieving performance that is comparable to that of fused logits without incurring additional inference time. We evaluate the effectiveness of our methodology across four distinct tasks. Our PED consistently demonstrates superior performance in all assessments when contrasted with numerous state-of-the-art methods. Moreover, our method can be seamlessly integrated into existing prompt learning approaches and consistently improves their performance. Our code is publicly available at https://github.com/vim-wei/PED.