Efficient Tuning of Multilingual ASR for Code-Switching

In a world where linguistic diversity plays a crucial role in communication, an increasing number of people are becoming polyglots-individuals fluent in multiple languages. As they engage in daily conversations, it's common for them to effortlessly switch between languages within a single sentence, a phenomenon known as code-switching. In the field of audio translation, this creates a distinct challenge. Recently, large pretrained multilingual speech models have demonstrated considerable potential in scaling Automatic Speech Recognition (ASR) to other languages where the datasets are sparse. Some of these models incorporate language adapters to enhance monolingual performance and mitigate some of the challenges associated with multilingual modeling, particularly for resourcerich languages. However, this approach limits their effectiveness in handling code-switched speech, where two languages are mixed within the same utterance. This research paper will explore how high-performing models like Whisper can be adapted for codeswitching through architectural modifications. These results could be valuable to the community when applying Whisper to tasks involving audio that frequently include code-switching within the target population. Impact Statement - Just as the Imagenet dataset sparked a momentum that led to the development of state-of-the-art image models, the research community is now focused on building multilingual ASRs that are crucial for understanding audio data and that can be applied within various business domains. Even the advanced multilingual ASRs face challenges when it comes to handling code-switching, which is common in countries where English is not the primary language of communication. The new techniques introduced in this paper reduce the Mean Error Rate (MER) by 70 % on the code-switching test dataset, making these models more suitable for domains with diverse, multilingual populations.