TavNet-An Audiovisual Speech Separation Model Based on Dual-Path Recurrent Neural Networks

Tracking a particular sound object in a mixed sound scene is a common skill for humans but remains challenging for machines. Traditional audiovisual speech separation models, such as frequency-domain methods and time-domain approaches, have demonstrated effectiveness but face limitations. Frequency-domain methods struggle with phase information modeling, while time-domain methods often encounter issues with long-sequence processing and parallelism. Moreover, prior models generally fail to fully exploit the complementary advantages of multimodal features, leading to suboptimal performance in complex acoustic environments. To address these challenges, this paper proposes TavNet, a novel time-domain audiovisual speech separation model that integrates visual features and dual-path recurrent neural networks (DPRNN). By incorporating lip motion features as robust visual inputs and utilizing the DPRNN architecture for efficient long-sequence modeling, the proposed model achieves significant improvements over audio-only and other time-frequency domain audiovisual methods. Experimental results on the LRS-2Mix dataset demonstrate that our model achieves an improvement of over 1dB and 3dB in SiSNR evaluation metrics for two-and three-speaker separation tasks, respectively, thereby establishing its superiority in real-world speech separation scenarios.