A Deep Learning-Based Strategy for Vietnamese Incorrect Pronunciation Detection

Despite the growing interest in learning Vietnamese, pronunciation remains a significant challenge for many language learners. This study explores the use of deep learning techniques to automatically detect incorrect pronunciation in Vietnamese. Our approach utilizes a multi-task setup that incorporates an Audio Encoder and a Phoneme Recognizer, enabling the model to learn the alignment between phonemes and acoustic features. This alignment information is then employed by the Incorrect Pronunciation Detector to identify words with incorrect pronunciation. Notably, we propose a novel strategy for generating pronunciation features, which involves “manually” grouping phonemes of the same word, thereby facilitating the model’s learning process. To evaluate the effectiveness of the proposed method, we build a small non-native (L2) Vietnamese speech dataset for training and testing. Compared to the baseline model, our final result improves the accuracy by 5.2% and $F_{1}$ score by 21.14%.