Multimodal Modeling of the Mora-timed Rhythm of Japanese and its Application to Computer-assisted Pronunciation Training

This paper discusses the setup of mobile components of StudyIntonation computer-assisted pronunciation training (CAPT) environment for pronunciation exercises for second language learners of Japanese. Grounded in signal processing, prosody-based speech graph visualization, and pitch quality estimation algorithms, the major focus of this contribution is on implementing the elements of multimodal pronunciation training for Japanese language, which serves as a model of a mora-timed language. We specifically address the approaches to CAPT feedback production with respect to representing mora-timed rhythmic patterns in a way that makes Japanese pronunciation training more enjoyable and more efficiently enables learners to master the mora-timed rhythm.