Modularized Pre-Training for End-to-End Task-Oriented Dialogue

Pre-training for
e
nd-to-end
t
ask-
o
riented
d
ialogue
s
ystems (EToDs) is a challenging task due to its unique knowledge base query (accuracy) need and lack of sufficient training data (fluency). In this paper, we try to mitigate the above challenges by introducing a modularized pre-training framework for EToDs, which achieves to effectively improve both accuracy and fluency of EToDs through a pre-training paradigm. The core insight is a modular design by decomposing EToDs into a
generation (fluency)
module and a
knowledge-retriever (accuracy)
module, which allows us to optimize each module by pre-training these two sub-modules with different well-designed pre-training tasks, respectively. In addition, such a modularized paradigm enables us to make full use of large amounts of KB-free dialogue corpus for the pre-training
generation
module, which can alleviate the insufficient training problem. Furthermore, we introduce a new
consistency-guided
data augmentation (CGDA) strategy to cope with the data scarcity problem to better pre-train the
knowledge-retriever
module. Finally, we fine-tune the pre-trained
generation
module and
knowledge-retriever
module jointly. Experimental results on three datasets show that our model achieve superior performance in terms of both fluency and accuracy. To our knowledge, this is the first work to explore modularized pre-training methods for EToDs.