Direct speech-reply generation from text-dialogue context

Natural speech-dialogue generation has been achieved with cascade systems combining automatic speech recog-nition, text-dialogue, and text-to-speech models. However, it is still challenging to generate expressive speech-replies depending on context because text-replies could lead to information loss in estimating appropriate expressions for speech generation. One promising approach is generating speech without requiring text. Direct speech generation from a dialogue context has never been achieved because it is difficult to learn the semantically one- to-many relationship between context and reply. This paper proposes a direct speech-reply generation model from the text-dialogue context in the same manner as the text-dialogue model. We focus on two challenges: an insufficient number of training dialogue pairs of text-context and speech-reply, and the difference between continuous speech signals and discrete text sequences. For the former, we applied text- to-speech to a text-dialogue dataset to acquire huge-scale training pairs. For the latter, we introduced the vector quantization on acoustic features to convert them into discrete sequences. The results indicate that the proposed model can successfully generate speech-reply directly from text-dialogue contexts, although a quality gap still exists with the text-dialogue model.