End-to-End Spontaneous Speech Recognition Using Hesitation Labeling

Spontaneous speech often contains hesitations, such as stammering, word substitutions, filler words and repetitions, unlike speech that is produced when reading aloud. These additional utterances create noise for Automatic Speech Recognition (ASR) systems, negatively affecting recognition accuracy. In this study, we propose an End-to-End (E2E) ASR system which can recognize these hesitations, as well as extra syllables, in spontaneous Japanese speech by training the ASR model using labeled data. Hesitation speech is then automatically labeled and ignored during speech recognition. Our experiments confirm that both the Character Error Rate (CER) and the Sentence Error Rate (SER) improved for all of the evaluation data sets in comparison to a baseline ASR method. In addition, when examining the actual recognition results, it was observed that the labels were inserted in the correct positions, suggesting that the model is able to correctly learn the meanings of the labels. Furthermore, by deleting the labeled utterances from there cognition results, we were able to obtain grammatically correct target sentences.