Voice Conversion With CycleRNN-Based Spectral Mapping and Finely Tuned WaveNet Vocoder

In this paper, we present a novel framework for a voice conversion (VC) system based on a cyclic recurrent neural network (CycleRNN) and a finely tuned WaveNet vocoder. Even though WaveNet is capable of producing natural speech waveforms when fed with natural speech features, it still suffers from speech quality degradation when fed with oversmoothed features, such as spectral parameters estimated from a statistical model. One way to address this problem is to introduce oversmoothed features while developing a WaveNet model. However, in a VC framework, providing oversmoothed spectral features of a target speaker for WaveNet modeling is not straightforward owing to the difference in the time-sequence alignment from that of a source speaker. To overcome this problem, we propose the use of a cyclic spectral conversion network, i.e., CycleRNN, capable of performing a conversion flow, i.e., source-to-target, and a cyclic flow, i.e., to generate self-predicted target spectra. The CycleRNN spectral model is trained using both conversion and weighted cyclic losses. To finely tune WaveNet, a pretrained multispeaker WaveNet model is optimized using the self-predicted features of the corresponding target speaker of a speaker conversion pair. The experimental results demonstrate that 1) the proposed CycleRNN-based spectral model for WaveNet fine-tuning significantly improves the naturalness of the converted speech waveforms, giving an overall mean opinion score of 3.50; and 2) the proposed model yields the highest speaker conversion accuracy, giving an overall speaker similarity score of 78.33%, which is a significant improvement compared with conventional WaveNet fine-tuning using natural target features.