W2N-AVSC: Audiovisual Extension For Whisper-To-Normal Speech Conversion

In this paper, we extend a method of converting speaking styles for whispered speech, i.e., whisper-to-normal speaking style conversion (W2N-SC). W2N-SC problem is similar but different from a regular voice conversion (VC) task and more challenging due to the characteristics of whispered speech and the deal with different speaking styles. In our previous study, we addressed the task-specific difficulties and developed a variational autoencoder (VAE)-based non-parallel approach called W2N-SC. While W2N-SC demonstrated superior performance to other parallel-data-free approaches, there remains room for improvement in conversion quality. To overcome the limitation, we propose W2N-AVSC, an audiovisual extension of W2N-SC. Unlike the conventional W2N-SC, the proposed W2N-AVSC can take visual information, e.g., lip movements, into account in the conversion of whispered speech. Furthermore, to perform W2N-AVSC, we develop a new audiovisual dataset recording the faces of speakers reading texts in various ways, such as in normals and whispers. Through experimental evaluations using clean and noisy whispered inputs, we reveal an effective representation of visual information, demonstrating that W2N-AVSC perceptually performs better than W2N-SC.