Towards Unseen Speakers Zero-Shot Voice Conversion with Generative Adversarial Networks

Zero-shot many-to-many voice conversion receives wide-spread attention but remains a challenging task. Recently, AUTOVC, which is based on conditional autoencoders, achieves state-of-the-art results in zero-shot voice conversion. However, the carefully designed bottleneck and the autoencoders based framework of AUTOVC limit further improvement of zero-shot voice conversion. In this paper, we propose a Generative Adversarial Network(GAN) based framework to disentangle the timbre and content of speech, and to synthesize new speech given unseen speakers and corpora. Towards unseen speaker, our framework extracts timbre embedding from an input speech with timbre encoder and produces content distribution embedding from any other speech with content encoder. Our framework learns to synthesize new speeches via conversion-reconstruction cycle training and to enhance the quality of conversion with adversarial training. Our experiments demonstrate that the proposed framework can generate outputs with comparable quality even for speakers that are not seen in the training dataset.