An Efficient Audio-visual Speech Enhancement Network via Multi-head Attention

T Audio-visual speech enhancement is regarded to be one of the promising solutions for isolating and enhancing the speech of the target speaker. Recent approaches mainly focus on estimating the speech spectrogram of the desired speaker via the convolutional neural network (CNN) with a naive concatenation operation, and these approaches ❨1❩ are unable to effectively capture the correlation between visual and audio modalities, and ❨2❩ cannot selectively process the features from these two modalities. To tackle these issues, we develop a visual speech enhancement network by using an encoder-decoder architecture with the multi-layer feature fusion strategy and acoustic and visual modalities balancing module. In particular, we apply ❨1❩ a modified multi-layer feature fusion strategy, which preserves each modality information while enjoying the benefits of the human auditory system, in which an early fusion strategy is combined with an intermediate fusion strategy, and ❨2❩ modalities balancing module, which contains a channel and spectral attention mechanisms via multi-head attention, which balances the information of two modalities in a learnable and self-adaptive manner. Experimental results demonstrate the superiority of the proposed model and suggest that the proposed multi-layer feature fusion strategy and modalities balancing module significantly improve the network performance.