Multiple-Embedding Separation Networks: Sound Class-Specific Feature Extraction for Universal Sound Separation

We propose a novel deep neural network (DNN) architecture for universal sound separation. This task aims to separate monaural mixture signals containing various sounds into the corresponding source sound signals (e.g., speech, barking of a dog, etc.). Previous studies used a speech separation network and made mixtures for datasets by sampling dry source signals randomly from a database. These methods did not use sound class labels, although they are available during the training step. We propose Multiple-Embedding Separation Networks, an architecture using sound class labels in a training step. This architecture contains multiple feature-extraction networks that are further specialized for each sound class. Each network is trained together with the class labels to capture characteristics as class-specific embeddings. We evaluated the performance of our proposed method and a method commonly used for universal sound separation as a baseline. This evaluation adopted a dataset containing six classes. As a result, the proposed method outperformed the baseline in terms of the average separation performance by 0.22 dB, especially for speech mixtures by 2.28 dB. We found a complex relationship between the amounts of the data of the dry source signal of each class and the separation performance.