Extending Whisper for Korean-English Code-switching Speech Recognition

Among various automatic speech recognition (ASR) models, Whisper is a state-of-the-art model that demonstrates robust performance across various multilingual tasks. However, Whisper shows poor performance in code-switching speech where multiple languages are used within a single sentence. To address this limitation, we propose an extended version of Whisper that utilizes a language detector model with a modified token sequence format. Unlike the original Whisper's token sequence, multiple language tokens can be assigned to one sentence using information from the language detector, which enables Whisper to effectively process code-switching speech. Our experiments on a one-hour Korean-English code-switching speech dataset showed a 1.7% reduction in the character error rate (CER) and a 0.9% reduction in the space-normalized word error rate (sWER) with only a 1.12 × higher relative time complexity compared to the original Whisper.