Multi-task Learning for Detection, Recovery, and Separation of Polyphonic Music

Music separation aims to extract the signals of individual sources from a given audio mixture. Recent studies explored the use of deep learning algorithms for this problem. Although these algorithms have proven to have good performance, they are inefficient as they need to learn an independent model for each sound source. In this study, we demonstrate a multi-task learning system for music separation, detection, and recovery. The proposed system separates polyphonic music into four sound sources using a single model. It also detects the presence of a source in the given mixture. Lastly, it reconstructs the input mixture to help the network further learn the audio representation. Our novel approach exploits the shared information in each task, thus, improving the separation performance of the system. It was determined that the best configuration for the multi-task learning is to separate the sources first, followed by parallel modules for classification and recovery. Quantitative and qualitative results show that the performance of our system is comparable to baselines for separation and classification.