Enhancing Video-Only Sign Language Recognition with Wasserstein Knowledge Distillation on the VSL Dataset

Practical Sign Language Recognition (SLR), especially for low-resource languages like Vietnamese Sign Language (VSL), is often hindered by the complexities of multi-modal systems requiring sensor data. To address this, we propose a video-only SLR solution leveraging knowledge distillation. Our framework transfers spatio-temporal knowledge from a powerful pre-trained Vision Transformer teacher (VideoMAE) to a lightweight Inflated 3D ConvNet (I3D) student using only RGB video for both student training and inference. A key innovation is the use of a Wasserstein Distance-based distillation loss, which proves more effective than traditional Kullback-Leibler divergence in guiding the student model. Extensive experiments on the VSL dataset demonstrate that our approach significantly outperforms various established baselines, including CNN-based for video (RGB-I3D), multi-modal fusion approaches (STSLR), motion-augmented streams (MARS), and Transformer-based models (SWIN-B) on the VSL dataset, and a strong multi-modal baseline (which requires both video and sensor data for training) and standard distillation techniques. Qualitative t-SNE visualizations further confirm the student’s enhanced ability to learn discriminative features. This work highlights a path towards more practical and accessible video-only SLR systems through effective knowledge transfer.