Noise and Reverberation-Controllable Voice Conversion

Recent developments in voice conversion (VC) systems aim to improve speech quality and speaker similarity even under challenging background factors, such as noise or reverberation. Although these factors have traditionally been viewed as unwanted interference, they can also provide valuable information in certain applications like movie dubbing. In this paper, we present a new approach to VC that allows for better control over background factors like noise and reverberation. Our method is capable of generating both clean and noisy-reverberant converted speech. During inference, we first enhance interfered input speech data using speech enhancement models to reduce noise and reverberation. We also use a denoising model and a reverberation time (T60) estimator to extract important background information from the interfered speech data, which is then used alongside a target speaker’s code to generate converted speech with the background factors. Our experiments show that the use of speech data with various acoustic conditions during training leads to improved speech quality and speaker similarity, even when clean data are not available. Moreover, optimizing the model with clean and pseudo-clean data further enhances the results.