Optimising Custom Models for Emulating Human Speech Synthesis in Tonal Languages

Over the past few years, personalised human speech imitation has gained significant attention due to advancements in deep learning technologies. However, there is still a noticeable difference between raw and imitated audio, especially in tonal languages. The challenge of accurately reproducing vowel quality poses a unique limitation, as it requires replicating nuanced pronunciation. Although speaker adaptation is a traditional method to address this issue, it often requires extensive datasets, making it costly. This study focused on Mandarin and proposed a personalised, optimised model for imitating human speech synthesis in tonal languages. Using a low dataset corpus, the model could support Few-Shot Learning (FSL) optimisation. This research not only pushes the boundaries of speech imitation technology but also offers valuable insights for further advancements in the fascinating realm of human speech synthesis. Our methodology involved implementing the mockingbird real-time clone application, which is considered the most proficient real-time speech cloning tool available on the market. We employed FSL to create personalised, optimised models using a minimal set of labelled vowel syllables. The proposed model is an optimisation layer that effectively enhances the naturalness of simulated human voices in tonal languages, even with limited dataset availability. This innovative approach significantly improved the similarity between Mel spectrogram imitated speech and raw speech, which increased from 53.8% to an impressive 68%. The model adds an optimising layer to improve the naturalness of simulated human voices in tonal languages with low datasets. To our knowledge, this marks the first successful attempt at imitating Mandarin speech with such fidelity.