SFMM: Semantic-to-Frame Matching with Multi-Classifier for Few-shot Action Recognition

Benefiting from the rapid development of large-scale text-image comparison models, they are also widely used in corresponding downstream tasks. In traditional action recognition tasks, the typical approach is based on neural network models for prototype comparison. However, this approach limits the models’ ability to migrate to unseen or few-shot datasets and faces problems such as insufficient data and depth information. To enhance the accuracy of the few-shot action recognition task, we propose the SFMM model. This model exploits the generalization performance of the CLIP model, which is trained on a large amount of data and complements the image features with semantic information. To circumvent the randomness in model training, we employ a multi-model stacking approach to enhance recognition. Specifically, we model the task as a text-video-action category-matching problem containing different modal information. The model’s main features include the introduction of a text-matching module for full semantic coverage and an improved feature enhancement module that fuses multimodal information through convolutional and cross-attention networks. These methods allow the model to fully utilize multimodal information and obtain highly discriminative category prototypes. Our model was tested on commonly used benchmarks, demonstrating the effectiveness of our proposed method and achieving excellent performance in action recognition tasks.