Contextual Aligned Tamil Sign Language Retrieval System with Semantic Masked Transformer

The sign language retrieval system, while similar to conventional video retrieval systems, faces distinct challenges that require a thorough understanding of the meanings conveyed through human actions depicted in the videos. Typically, meaningful information is extracted using RGB feature extraction techniques. However, this method can lead to redundancy, as it often fails to capture the global contextual information inherent in each video frame. Additionally, the reliance on RGB features can present significant memory challenges due to the large data requirements associated with processing and storing this information. Consequently, a more efficient approach is necessary to enhance both the accuracy and effectiveness of sign language retrieval. To address these issues, we have implemented a semantic masked transformer architecture specifically tailored for the sign language retrieval task. This innovative system not only incorporates RGB features but also integrates pose features, which are derived from the movements and positions of the signer. By fusing these two types of features based on their similarity, the system can provide a more refined and pertinent retrieval of textual information related to the video frames. Our extensive experiments conducted on a custom dataset designed for Tamil Sign Language demonstrate that this framework significantly outperforms current state-of-the-art methods. These results underscore the effectiveness of combining semantic understanding with advanced feature extraction techniques, ultimately leading to improved accuracy and efficiency in sign language retrieval.