DecoupledSynth: Enhancing Zero-Shot Text-to-Speech Via Factors Decoupling

Studies of speech representation enhance zero-shot Text-to-Speech by mapping text to intermediate representations before generating speech. However, using representations often struggles to balance linguistic, para-linguistic, and non-linguistic information in speech during the synthesis phase. Additionally, it usually takes substantial resources for representation extraction training. To address these limitations, we propose DecoupledSynth. It combines different self-supervised models to extract comprehensive, decoupled representations. This structure enables more thorough and nuanced synthesis by leveraging reference speech with decoupled processing stages. Experiments on the VCTK and LibriTTS datasets support the potential of this new framework, showing that it can produce more consistent and realistic speech. Speech demos are available at https://test1634.github.io/DecoupledSynth/.