SynthoTranslate: Multimodal Knowledge Assimilation Tool for Overcoming Auditory Barriers and Bridging Language Barriers

This paper introduces an automated system to improve educational accessibility through multilingual video translation and dubbing. It uses advanced speech-to-text (STT) technologies, natural language processing (NLP), and voice conversion models for accurate subtitles, translations, and dubbed audio in Indian languages. Key components include Whisper and Wav2vec for subtitle generation, GoogleTrans for translation, and Silero models for natural-sounding voice cloning. Indian Sign Language (ISL) transcription is integrated to support hearing-impaired students, with Wav2Lip ensuring precise lip-syncing of dubbed audio. Evaluation shows a Word Error Rate (WER) of 1.9% and cosine similarities over $\mathbf{9 5 \%}$. The system also offers content summarization and Q&A to enhance engagement. Future work will focus on optimizing speech recognition and improving ISL transcription for greater accessibility.