The use of a formant diagram in audiovisual speech activity detection

We present an audiovisual approach to the problem of voice activity detection for systems with a single microphone and a single camera with multiple people in the camera's field of view. We aim to have a speech activity detection result per person. The approach utilizes a face tracking and lip contour tracking algorithm for the video analysis, and pitch presence detection and formant frequency tracking algorithms for the audio analysis. When from the audio we detect speech activity and from the video we find lip activity for more than a single person, we check for each person whether the vowels correspond with the video mouth parameters to find out if this person speaks. To this end we make use of the F
₁
-F
₂
speech formant diagram in which we propose three vowel groups that are distinctive both from audio and video data.