Noise-Tolerant Time-Domain Speech Separation with Noise Bases

Time-domain Audio Separation Network (TasNet) is a deep-learning-based speech separation system that performs end-to-end learning in time domain. By directly using speech signal in a time domain, TasNet outperforms conventional systems based on a time-frequency domain in terms of reconstruction quality and latency. In TasNet, the separation accuracy is greatly degraded when speech contains noise. We propose TasNet with noise basis signals (TasNet-NB), a method to improve separation performance under noisy environments by adding noise basis signals to speaker's basis signals. It utilizes curriculum learning to gradually reduce the noise signal-to-noise ratio and the noise reconstruction loss as one of the objective functions in training. We evaluate the method on WHAM! dataset and show that it improves SI-SDRi from 13.7 dB to 14.6 dB.