Cross-modal Speech Separation Without Visual Information During Testing

Visual information plays an important role in speech separation. It has been illustrated by many studies that audio-visual speech separation has better performance than audio-only speech separation. However, it is not always possible to capture high-quality visual information due to various factors. In this paper, we propose a novel audio-visual matching-based speech separation method. The network exploits audiovisual correlations in the training process, but does not require visual information during testing. The proposed cross-modal speech separation method is evaluated on GRID dataset. Experimental results show that the proposed method has a significant improvement in multiple evaluation metrics for different types of mixtures compared with other advanced speech separation methods.