A Simple Method to Improve the Performance of Small Pre-trained Language Models on Few-shot Tasks

The goal of few-shot learning is to train a model with good generalization ability when the training data is limited in quantity. Prompt Learning-based methods have become the mainstream solution for addressing such problems. In the current Prompt Learning methods using small pre-trained language models(with parameters under 1B), the Verbalizer component faces challenges such as limited answer search space or inadequate answer search guidance. To address these issues, this paper proposes Contrastive Prompt Learning (CPL) to optimize answer search. In our proposed method, the manually designed Verbalizer is replaced by a linear layer, which expands the search space for answers. While this allows for more accurate answers, it also potentially increases the difficulty of searching for answers. Therefore, this paper introduces Contrastive Learning to guide the model in searching for answers. Across multiple few-shot learning tasks, our approach compared against other powerful baseline methods, and the approach has achieved significant improvements. In the current prevalence of Large Language Models, this paper also conducted comparative experiments by fine-tuning chatglm2-6B, and the experiments indicate that our approach still maintains advantages in many tasks.