An Audio-Visual Separation Model Integrating Dual-Channel Attention Mechanism

Sound source separation is the separation of targeted sounds from a noisy environment, which plays an important role in signal processing and has been studied extensively. However, most of these researches only extract audio information for processing, ignoring visual information, resulting in the waste of feature information. In addition, some researchers have fused the extracted features but have not noticed the weights of different features, resulting in a poor model effect. This paper uses multi-modal information to separate sound sources to solve these problems. We constructed a multi-modal Audio-Visual separation model integrating the Dual-channel Attention mechanism named AVDA. The model realizes sound source separation through the dynamic fusion of visual and audio features. Specifically, firstly, we take audio video as input for data preprocessing and segment it to obtain video frames and audio. Then they are introduced into the visual and audio feature extractors, which integrate the attention mechanism to extract features. Finally, the extracted visual and audio features are introduced into the fusion prediction component to obtain the predicted spectrogram, which is compared with the ground truth spectrogram for subjective evaluation. In addition, three indexes of signal distortion ratio (SDR), signal interference ratio (SIR) and signal artifact ratio (SAR) are introduced for quantitative comparison. The experimental results on MUSIC-21 data set show that the model achieves 10.96, 17.91 and 12.77 respectively in three indexes, and its performance is significantly better than other models in audio-visual separation tasks.