Automatic Translation of Indonesian Sign Language Based on Multimodal Technology Recognition

Multimodal technology integrates inputs such as audio, video, and text to enhance natural language-based communication accessibility for individuals with disabilities. In Indonesia, people with hearing impairments continue to face challenges in accessing information due to limited sign language interpreter availability and the lack of contextual and interactive learning platforms. This research aims to develop a web-based translation platform that converts Indonesian audio, text, and video into BISINDO gestures to support inclusive communication. The proposed system uses speech recognition, NLP-based text preprocessing using tokenization, normalization, and stemming (via Sastrawi), and audio extraction using MoviePy, transcription with Speech Recognition, and gesture mapping using Supabase-based dataset with synonym handling and fallback to fingerspelling and all integrated into React-Flask architecture. Experimental results demonstrate strong performance: speech transcription achieved 95.1 % accuracy with a 3.68 % WER, text-to-gesture translation reached a 77.27 % F1-score on first-time inputs (71 seconds) and 100 % on repeated tests (1 second), while video translation improved from 13-20 % to 100 % accuracy, with processing time reduced from 14-19 minutes to just 2-4 minutes through gesture caching confirming the system's efficiency, adaptability, and potential for accessibility and learning.