Toward Visual Pronunciation Learning: A Speech-to-Articulatory Animation Pipeline Leveraging wav2vec 2.0 and rtMRI Landmarks

Most computer-assisted pronunciation training (CAPT) systems for second language (L2) learners focus on detecting mispronunciation based on predefined phonemes and assigning pronunciation scores. However, these systems often lack visual feedback or detailed corrective guidance, limiting learners’ opportunities for significant improvement. This paper presents a key advance toward developing a CAPT system that offers detailed visual feedback on articulatory movements using real-time magnetic resonance imaging (rtMRI) articulatory landmarks. The limited availability of paired speech and articulatory landmark data, typically involving only a few speakers, poses a challenge for generalizing across diverse speech patterns. To address this, we propose leveraging pretrained wav2vec 2.0 embeddings, fine-tuned to generate articulatory contours mapped to xy coordinates based on rtMRI landmark data. As evaluated with the rtMRI USC-TIMIT dataset, our system effectively reconstructs visual articulatory movements from speech, marking a significant step toward enhanced visual pronunciation learning.