M³-SLR: Self-Supervised Pretraining With MaxFlow MaskFeat for Improved Multi-View Sign Language Representation

Sign Language Recognition (SLR) is crucial for facilitating communication with the hard-of-hearing community, yet remains significantly challenged by factors such as inter-signer variations, self-occlusion, and the critical issue of Visually Indistinguishable Signs (VISigns) – signs appearing identical from a single viewpoint. While multi-view systems offer potential solutions by providing richer spatial information, effectively leveraging this data, particularly through pretraining tailored for fine-grained sign dynamics, remains an open problem. Furthermore, the computational demands of multi-view models hinder practical single-camera deployment. To address these limitations, we introduce M3-SLR, a comprehensive framework for Multi-View Isolated Sign Language Recognition (MultiView-ISLR). M3-SLR integrates three key proposals: 1) A novel MaxFlow Cube Masking strategy for MaskFeat, a self-supervised pretraining method, which utilizes optical flow to guide spatiotemporal cube masking, compelling the model to learn representations focused on dynamically critical regions essential for distinguishing signs. 2) The UF3V multi-view architecture, built upon effective UniFormer backbones and incorporating a Dual Co-Attention mechanism for the fusion of features from three adjacent views, significantly enhances the ability to resolve ambiguities like VISigns. 3) An efficient knowledge distillation process called 3-view to 1-view, which transfers the learned multi-view knowledge into a high-performance single-view model, bridging the gap to practical application while maintaining the ability to distinguish VISigns. Extensive experiments conducted on challenging public MultiView-ISLR benchmarks, Multi-VSL200 and MM-WLAuslan, demonstrate that M3-SLR achieves state-of-the-art performance using only RGB data, validating the effectiveness of our proposed framework for robust and practical SLR. Our code is available at https://github.com/fossbk/M3-SLR.git