Prompt-Based Few-Shot Text Classification with Virtual Class Representation

Over the past few years, "Pre-training + Fine-tuning" has been highly successful in Natural Language Processing (NLP) tasks, but it has limitations in few-shot tasks and zero-shot tasks. Recently, "Prompt-based Tuning + Pretrained Language models (PLMs)" has achieved great success in few-shot stage and zero-shot stage, including few-shot text classification. Prompt-based tuning wraps suggestive characters and [MASK] tokens into a raw input sequence. Additionally, Prompt learning uses a verbalizer to map the semantic information of the [MASK] position to the final classification prediction. The verbalizer contains manual verbalizer and automatic verbalizer, etc. However, manual verbalizers are too dependent on human experience and external knowledge, while automatic verbalizers still face significant challenges. In this work, we introduce a novel approach for constructing a verbalizer termed as “virtual class representation”. This method entails assigning a fixed-length sequence of digits to represent each label in the dataset. We train a small-scale randomly initialized embedding mechanism to achieve preliminary semantic information for each class when virtual class representation is introduced into the embedding mechanism. The semantic information of each category is then further extracted by a convolution operation, and the final model regression value is obtained by calculating the similarity between the last hidden state of PLM and the semantic vector of the class. Our extensive experiments demonstrate the effectiveness of the proposed method on few-shot text classification tasks.