Open-Vocabulary Action Recognition with Masked Visual Prompt and Verb Semantic Reconstruction

Open-vocabulary action recognition aims to recognize actions beyond a predefined set of action categories, leveraging vision-language models for improved generalization. However, existing methods often suffer from static scene bias and limited understanding of motion information, leading to reduced accuracy for both known and novel categories. To address these challenges, this paper proposes MVPR, a novel open-vocabulary action recognition method based on masked visual prompts and verb semantic reconstruction. First, we introduce a Mask-guided Action-Background Decomposition (MABD) module, which utilizes segmentation masks as visual prompts to decompose the action and background regions, thereby reducing the interference of static scene information. To further leverage contextual background information without introducing biases, we propose a Selective Feature Communication (SFC) module that adaptively integrates useful background information into the action representation. Additionally, we design a Verb Semantic Reconstruction Loss to enhance the model's capability in capturing fine-grained motion features. Experimental results on HMDB51, UCF101, Kinetics400, and Kinetics600 demonstrate the effectiveness of our MVPR method.