Automatic Generation of Diagnostic Content Feedback in Spoken Language Learning and Assessment

This study aims to explore an effective way to automati-cally generate actionable diagnostic feedback targeting the content development skill within the context of spoken language learning and assessment. There is one type of test question widely used in speaking assessment, which requires test takers to first listen to and/or read stimulus material and then create a spontaneous response to a question related to the stimulus. In a high-proficiency response, critical content from the stimulus - referred to as “key points” - should be properly covered. We propose Transformer-based models to automatically detect the location of key point spans within a response, or if no key points are covered, to detect their absence. Additionally, we introduce a multi-task learning approach for assigning a “quality score” to each key point span, measuring how well the key point is rendered within the response. Experimental results demonstrate that the pro-posed models can surpass human expert performance based on human transcriptions. In contrast, based on automatic speech recognition hypotheses, models underperform human expert performance with an F1-score at 66.6% (vs. human agreement of 68.8%) on span detection and a Pearson corre-lation coefficient at 0.579 (vs. human agreement of 0.610) on quality score prediction.