HAPX-CLIP: Human Activity Recognition with Visual Sequences and Language Prompts

Video-based Human Activity Recognition (VHAR) is a core task in computer vision with a wide range of applications in healthcare, surveillance, and human–robot interaction. Traditional VHAR methods usually extract visual features from frames and predict the category of the activity. However, these methods often lack semantic and interpretability. With the rise of foundation language models, multimodal image-language approaches have shown that language prompts can significantly enhance image-based tasks. However, applying such prompts to video recognition remains underexplored. A recent method, X-CLIP, improved video recognition by incorporating class labels as textual prompts, but it provides limited semantic detail. In this paper, we build on X-CLIP by introducing HAPX-CLIP — a Human Activity Prompt-extended X-CLIP — which uses action-conditioned prompting to generate more descriptive and semantically rich text prompts. This enhances vision-language alignment and improves interpretability. We evaluate HAPX-CLIP on three HAR benchmarks: MuWiGes, UESTC, and MMAct. HAPX-CLIP consistently outperforms X-CLIP, achieving a notable Top-1 accuracy improvement of 9.76% on the challenging MuWiGes dataset. These results confirm the effectiveness of using semantically enriched prompts for scalable and interpretable vision-language-based HAR.