Transformer-based Automatic Speech Recognition of Simultaneous Interpretation with Auxiliary Input of Source Language Text

Automatic speech recognition (ASR) of simultaneous interpretation is challenging due to disfluencies such as hesitations, filled pauses, interruptions, and self-repairs. Meanwhile, the simultaneous interpretation sometimes accompanies source language text. In particular, training materials for simultaneous interpreters have the source language text. The source language text can also be considered an ideal transcription of source language speech. Previously, we proposed a Transformer-based ASR with auxiliary input of source language text for transcribing simultaneous interpretation and verified the effectiveness of the model structure by using large-scale corpora of speech translation (ST). In this study, we fine-tuned and evaluated the proposed model using a corpus of simultaneous interpretation with transcription of the source language speech as the auxiliary input. We also added a filled pause model to the ASR and compared the effect on the reduction of a word error rate (WER) with the auxiliary input. The experimental results showed significant WER reductions by both the auxiliary input and the filled pause model, though the reduction made by the filled pause model was greater than that made by the auxiliary input.