Rethinking Singing Voice Separation With Spectral- Temporal Transformer

Recently, singing voice separation from polyphonic music using fully convolutional neural networks (CNN s) has achieved promising performance. Polyphonic music often has a long-term dependency and wide frequency bands. Therefore, a large receptive field of CNNs is critical for singing voice separation. However, the fundamental concerns still raise because of the limited receptive field of CNNs. Besides, the intrinsic spectral-temporal characteristics in the spectrum are neglected by the traditional CNNs. This paper aims to provide a rethinking of model design and proposes a pure spectral-temporal transformer-based encoder to replace CNNs as an alternative. Further, we propose an attentive fusion module to attend to spectral and temporal features and fuse them dynamically. The fused features are then fed to the decoder to obtain the final mask. Experimental results show that the proposed model outperforms several existing well-known methods.