Sign, Attend and Tell: Spatial Attention for Sign Language Recognition

Sign Language Recognition (SLR) has witnessed a boost in recent years, particularly with the surge of deep learning techniques. However, most existing methods do not exploit the concept of attention mechanisms, despite their success in several computer vision tasks. In this paper, we propose a novel method for isolated SLR which utilizes spatial attention to focus the processing on the informative, discriminating parts of the input. This is particularly important for SLR, since the RGB image contains several distracting information such as background and signer's clothes, which are irrelevant for the task. We investigate three ways for incorporating spatial attention: a) pre-focused attention, which uses optical-flow-based motion as a prior b) learned attention, where the network learns where to focus during training, and c) hybrid attention, which combines both approaches by initializing the attention layer in the learned attention with the motion-based attention masks used in the pre-focused attention. We show, first, that all three approaches outperform state-of-the-art methods on one of the largest isolated SLR datasets, validating the effectiveness of attention mechanisms on the SLR task, and second, that the best performing approach is the hybrid attention, combining both ideas.