Modelling the tongue movement of Chinese Shaanxi Xi'an dialect speech

Visual speech play a significant role in speech perception especially for deaf and hard of hearing people or normal people in noisy environment. Lip reading depends on visible articulators to improve speech perception. However, not only the movement of lip and face provide part of phonetic information, but also the motion of the tongue which is generally not entirely seen carries an important part of the articulatory information not accessible through lip reading. First, static viseme (visual representation of a phoneme) classification of Chinese Shaanxi Xi'an dialect speech was performed according to the method that carried out to classify Mandarin (Standard Chinese) static viseme. Then we carried out an experiment with the purpose to study both the timing and position properties of articulatory movements of the tongue in VCV and CVC utterances of Chinese Shaanxi Xi'an dialect speech in different tempo. The speech materials conclude VCV and CVC sequences were recorded via `Micro' ultrasound system that eventually form the JPG images and MP4 videos with the help of Assistant Advanced software. We have developed an algorithm that can automatically carry out a spatial-temporal tracking of tongue movement contour from the ultrasound images we recorded of one speaker who read randomized lists of VCV utterances containing the vowels /e, a/ and CVC utterances containing the consonants /k, t/ in all possible combinations covering all phoneme - 26 consonants and 41 vowels of Shaanxi Xi'an dialect. The extracted visual information gained by this method is then classified in order to define the uttered viseme and is used to create dynamic viseme system of tongue for Shaanxi Xi'an dialect. This dynamic viseme system is used to create the fundamentals of a talking head - animated articulation model for Shaanxi Xi'an dialect speech. The talking head is going to be utilized in a speech assistant system assisting the speech improvement of hard of hearing children and training for second language learners.