Towards A Vocalization Feedback Pipeline for Language Learners

Practice is essential for language learning. This is true for writing as well as speaking. However, in contrast to writing, it can be challenging to offer students sufficient time to practice speaking while receiving corrective feedback from a teacher. Considering the importance of corrective feedback for language mastery, automatic feedback systems could provide vital assistance through additional supervised speaking exercise. For this reason, this paper proposes an end-to-end feedback generation pipeline to correct grammar errors in unconstrained speech. The approach consists of four steps, converting raw speech files into transcripts with automatic speech recognition, removing disfluency, correcting grammatical errors, and preparing the feedback for presentation to the user. An explorative analysis of the pipeline with English language learners indicates that out-of-the-box automatic speech recognition models degrade in performance when used by language learners. However, training the model with only 15 minutes of learners’ speech decreases the word error rate almost by half.