Perspective Makes Perfect: Prompt-tuning Vision-Language Models for Action Recognition with Diversified Multi-Modal Observation

Observing key visual cues and reasoning multi-modal semantics from diversified perspectives are critical in human beings’ perception of actions. However, existing action recognition methods, including ones based on prompt-tuning image-based vision-language (I-VL) models, suffer from the under-exploration of video contents and textual semantics. In this paper, inspired by how human beings reason actions from diversified perspectives, we propose Diversified Multi-Modal Observation (DMMO) to improve prompt-tuning for video action recognition by reducing redundancy and enhancing semantics diversity in various perspectives. Firstly, we select a few informative frames instead of dense sampling for clear visual semantics. Then, subject-context segmentation is applied as diversified visual emphases for textual semantics analysis. Afterward, auxiliary captions with diversified perspectives are generated and aggregated. Finally, we perform multi-modal interaction and align outputs. Extensive evaluation experiments show the superiority of our proposed method, and demonstrate the effectiveness of diversifying prompts from multiple perspectives in tuning I-VL models.