Listening to Each Speaker One by One with Recurrent Selective Hearing Networks

Deep learning-based single-channel source separation algorithms are currently being actively investigated. Among them, Deep Clustering (DC) and Deep Attractor Networks (DANs) have made it possible to separate an arbitrary number of speakers. In particular, they cleverly combine a neural network and a K-means clustering algorithm to obtain source separation masks with the assumption that the correct number of speakers at the test time is known in advance. Unlike DC and DAN, Permutation Invariant Training (PIT) was proposed as a purely neural network-based mask estimator. Essentially, however, PIT can deal with only a fixed number of speakers, given the strong relationship between the dimensions of the output nodes and the assumed number of sources. Considering these limitations and merits of such conventional methods, this paper proposes a purely neural-network based mask estimator that can handle an arbitrary number of sources, and simultaneously estimate the number of sources in the test signal. To accomplish this, while the conventional methods deal with the source separation problem as a one-pass problem, we cast the problem as a recursive multi-pass source extraction problem based on a recurrent neural network (RNN) that can learn and determine how many computational steps/iterations have to be performed depending on the input signals. In this paper, we describe our proposed method in detail, and experimentally show its efficacy in terms of source separation and source counting performance.