Pure Voice : Efficient Speech Separation with Audio-Visual Attention

We introduce a novel audio-visual speech separation network. When dealing with a video containing multiple speakers and background sounds, our primary objective is to extract speech segments associated with specific identities. Current approaches predominantly rely on the correlation between the speaker's lip motion features and their voice features. In contrast, we leverage facial features as additional prior information to predict speech characteristics. Simultaneously, we have developed an audio-visual attention (AVA) module to concentrate on the visual features most relevant to speech separation. Our approach allows for the joint learning of audio and video features, as well as cross-modal embedding in unlabeled videos. It demonstrates superior performance on benchmark datasets.