DCF-Net: Efficient Target Speaker Extraction by Leveraging Mixture and Enrollment Interactions

Target speaker extraction (TSE) aims to isolate a specific speaker’s voice from multi-talker environments using enrollment data. While current approaches primarily utilize speaker embeddings from enrollment, they often neglect contextual information and the dynamic interactions between the mixture and enrollment. To address this limitation, we propose a novel DualStream Contextual Fusion Network (DCF-Net) that operates in the time-frequency (T-F) domain. Our framework introduces a DualStream Fusion Block (DSFB) that: 1) captures contextual information, 2) models interactions between contextualized enrollment and mixture representations across spatial and channel dimensions, and 3) employs these enriched representations to guide the extraction process. Comprehensive experiments show that DCF-Net achieves state-of-the-art (SOTA) performance with a 21.6 dB improvement in scale-invariant signal-to-distortion ratio (SI-SDR) on benchmark datasets while demonstrating robustness in noisy and reverberant conditions. Notably, our model significantly reduces the wrong extraction rate to just 0.4% when testing on target confusion problem (TCP), underscoring its practical applicability.