Multi-modal speech recognition using correlativity between modality

In recent years, to achieve robust speech recognition against noises, Audio-Visual Speech Recognition(AVSR) system utilizing not only audio but also visual information of lip has been studied. This paper proposes a decision method of the weight called stream exponent representing reliability ratio of audio and visual features. The method focuses on the correlation between audio and visual modality in order to estimate the optimal stream exponent. Furthermore, we modified the stream exponent using periodicity of speech, such as pitch, to handle abrupt noises. An audio-visual database is comprised of specific speaker's lip image sequences and audio sequences. The contents of the utterance are Japanese counting numbers and sound-alike words. Using this database we constructed the AVSR system and performed an evaluation experiment. The obtained results verify the availability of the proposed method under a variety of noisy environment.