Real-Time One-Shot Voice Conversion Based on High-Order Recursive Networks

There is always a high demand for voice conversion in the fields of entertaining dialogue, pronunciation correction, even voice cloning scams, etc. In the early stage, voice conversion relied on a model-based information bottleneck to maintain the speech content and morph the voice style. This approach, however, often leads to the deterioration of essential information, which in turn causes inferior speech quality and ineffective content reconstruction. Subsequent research has shifted towards using digital signal processing (DSP) to modify the style of the source voice while preserving the content by assimilating the necessary style from supplementary data. As a significant advancement in deep learning, several technologies, such as recursive networks and generative adversarial networks, have become a growing trend in voice conversion. As a result, this study presents a real-time voice conversion model implemented on the Nvidia Jetson AGX Xavier platform. It uses the multilayer architecture of HorNet to thoroughly analyze voice features with the assistance of conditional instance normalization (CIN) to efficiently adjust the target style in the perturbed input voice. Additionally, the model utilizes log mel-spectrograms as the representation for training. Doing so not only improves the efficiency of processing voice data but also keeps frequency features that are generally more perceptible to human hearing. The following vocoder, HiFi-GAN, also ensures a high-quality latent feature to waveform transformation. Moreover, objective assessment confirms that the synthesized voices outperform relevant research in terms of reconstruction accuracy and processing time.