A fuzzy synchronization algorithm for bimodal speech signals

This paper describes a rule-based fuzzy system that estimates the relationship between acoustic and visual speech and uses this estimate for the synchronization of not aligned audio-visual signals. The relations are quantified by means of a set of rules, which associate typical mouth shapes (visual classes) to specific acoustic classes. The visual and acoustic classes are learned from training data using automatic clustering algorithms, relying on the clustering tendency of the extracted feature vectors and without performing phonetic recognition. Nevertheless, the categorical fuzzy structure of the system allows one to recognize, with some degree of uncertainty, the phonetic and visematic characteristics of the speech signal, making thus possible the integration of other sources of information in a human-like way.