Multilingual Video Dubbing System

This research introduces a web-based video dubbing system for multilingual Indian languages, employing advanced transformer models. The system transcribes, translates, and dubs videos into different languages using a combination of Open AI Whisper for audio transcription, Facebook MBart for text translation, and Silero for text-to-speech conversion with multiple accents. To create a natural viewing experience, the platform integrates a lip sync module for precise alignment of facial movements with dubbed audio. Users can upload videos or provide youtube urls, then choose from various output options, including video-to-audio conversion, original video subtitles, or translated subtitles. Built with React, Python, and Flask, the system offers an efficient and scalable solution for cross-language video communication. Key challenges addressed include multilingual accessibility, speech recognition accuracy, translation quality and lip synchronization aiming to enhance the overall quality and accessibility of video dubbing for Indian languages.