Audio-Visual Single-Channel Signal Separation Based on Big Data Augmentation

Cocktail party problem has attracted more attentions in recent years in the speech community. Specially, the single-channel multi-talker speech separation and recognition has become a research hotspot. Also, the visual based information has also been adopted to improve the performance of speech separation and speech recognition. In this paper, we explored to improve the baseline permutation invariant training (PIT) based speech separation systems by two data augmentation methods. Firstly, the visual based information is selected to determine the permutation of separated speakers and also improve the separation performance. Besides, the SpecAugment was explored with big data augmentation method to improve the performance of separation. Finally, we achieved dB of SDR on a mixed dataset using TCD-TIMIT corpus.