On Off-Screen Speech Separation based on Deep Learning of Audio-Video Fusion

Speech separation with audio-video fusion becomes one of the most attractive topics in sound signal processing recently. However, most methods currently emphasize the onscreen sounds that are intimately connected to video information. In this paper, we propose a novel deep learning model that simultaneously pays attention to the on and off-screen speech for the task of separating on/off-screen speech. We applied audio-video fusion networks based on Convolutional Neural Network (CNN) for feature extraction and fusion of audio and video features. Secondly, we employed u-net to separate the spectrogram of sounds and enhanced the performance of on-screen speech separation by concatenating audio-visual fusion features. And then, in pursuit of superior off-screen sounds, we designed a residual u-net to generate residuals spectrogram that might be removed by mistake during the process of the off-screen spectrogram. Experiments show that the proposed model achieves enhanced performance compared with other works on the VoxCeleb2 dataset.