An Evaluation of the Use of Text-Based Comprehensibility Measures on Online Spoken Language Learning Materials

Automated assessment of online spoken materials for use in listening training can be of benefit to language learners. ICALL systems with this facility can help learners sift through collections of online content to find materials that suit their proficiency level and learning goals. Our pilot study assesses the extent to which readability measures can contribute to the listenability assessment of online spoken materials for language learning. Extending these text-based measures to assess listenability is convenient, but their use must be reassessed against spoken materials available online. We compare assessments by four readability measures with expert-assigned assessments of difficulty on online spoken language learning materials. We also evaluate the robustness of these measures against errors arising from automatically generating transcripts and their capability to predict human-based judgments of listenability. Our findings show that these comprehensibility measures can be combined and used to discriminate between materials of different levels of difficulty. We are also able to illustrate a way to conduct a fully automated approach to listenability assessment of spoken language learning materials via text-based measures. Our study also highlights the need for an approach that looks beyond the text, particularly for assessing higher-level materials and spoken materials of different genres.