Cross-Modal Adaptive Prototype Learning for Continuous Sign Language Recognition

Continuous sign language recognition technology enables effective communication for hearing-impaired individuals by recognizing and interpreting sign language. However, existing research has not fully addressed the large amount of temporal and spatial redundancy in sign language videos, which limits recognition accuracy. Additionally, the scarcity of frame-level annotated data hinders the widerspread utilization of gloss-level features from sign language videos by existing weakly supervised learning methods, thereby impedes the model from sufficient training. To address the above challenges, we propose a novel Cross-Modal Adaptive Prototype Learning model for Continuous Sign Language Recognition (CAP-SLR), which leverages prototype learning to fuse features across different modalities and improves recognition accuracy. Initially, we propose a lightweight Keyframe Extractor and a Multi-Scale Dilated Convolutional Attention to alleviate data redundancy and bolster the visual representation of sign language based on spatial-temporal information. Subsequently, we employ the Contextual Position Encoding-assisted (CoPE) transformer to learn the semantic of sign language, ameliorating the issue of cross-modal semantic prior bias inherent in pre-trained models. Finally, we design a Cross-Modal Adaptive Prototype Updating mechanism (CAP), which adaptively fuse visual features, gloss prototypes, and textual features, and subsequently update the gloss prototypes through the classifier-aware feature states, effectively mitigating the problem of easily introducing erroneous features in traditional momentum update methods. Extensive experiments on the PHOENIX14, PHOENIX14-T, and CSL-Daily datasets demonstrate that the proposed CAP-SLR method can effectively align cross-modal features at the gloss level, and achieves competitive recognition performance.