Cross-Modality Consistency Mining For Continuous Sign Language Recognition with Text-Domain Equivalents

Continuous Sign Language Recognition (CSLR) approaches share similarities with conventional NLP approaches in which language understanding is involved. However, CSLR approaches face the significant challenge of limited scale and vocabulary in existing datasets. Unlike language models that benefit from extensive training datasets, CSLR models often contend with data constraints, hindering their ability to generalize effectively and consistently capture the rich sign language expressions. To leverage the strong contextual and memorial capabilities of pre-trained language models, in this work, we propose Cross-Modality Consistency (XMC) loss to mine the alignment between the visual model and pre-trained language model, enabling the direct alignment at the gloss level. Towards this, we construct a small-scale gloss description corpus named DCSLG with rich descriptive text. Accompanied by CSL-Daily, text-domain equivalents are made for each video in the dataset, making fine-level alignment possible. The experiment results show that the proposed XMC Loss significantly improves the activations, producing more spatio-temporally accurate and relevant activations. Our approach achieves an average reduction in WER by 3.5%. The code and our gloss description corpus named DCSLG are made publicly available on GitHub ¹.