MetricCycleGAN-VC: Forcing CycleGAN-based Voice Conversion Systems to Associate with Objective Quality Metrics

In CycleGAN-based voice conversion (VC) systems, such as MaskCycleGAN-VC, similar to many other GAN-based VC systems, adversarial losses are not explicitly designed to optimize VC evaluation metrics. Consequently, the generator may not consistently improve the metric scores related to VC assessments, such as the Mel-Cepstral Distance (MCD). To address this, we propose MetricCycleGAN-VC, which optimizes discriminators based on one or more VC evaluation metrics in a black-box strategy. At the same time, users can specify their preferred and desired metrics. Our approach covers various metrics related to VC quality triple aspects: speaker similarity, speech naturalness, and intelligibility. Importantly, our approach maintains the integrity of these metrics. Unlike the conventional CycleGAN-VCs family members, MetricCycleGAN-VC trains generators and discriminators tied to specific evaluation metrics. We also leverage full-reference image quality or similarity metrics like Feature-based similarity index Measure (FSIM) to enhance speech quality indirectly. Within a unified framework for non-parallel voice conversion, we combine the optimization of classical intrusive speech quality assessment metrics (MCD, Short-Time Objective of Intelligibility (STOI), neural tools such as Non-Intrusive objective Speech Quality Assessment (NISQA), and a newly proposed metric called Speaker Embedding Cosine Similarity (SECS). Our method allows the utilization of existing or newly developed intrusive and non-intrusive quality metrics without compromising their structural integrity or requiring knowledge of their formulation. Experimental results demonstrate improved desired metrics in both inter- and intra-gender conversions, showcasing the superiority of our approach over MaskCycleGAN-VC a typical baseline and many state-of-the-art VC systems. One significant advantage of our proposed method lies in its adaptability to advancements in VC quality metrics. As more accurate metrics emerge and are applied, we can achieve even better results.