Modularized Pre-Training for End-to-End Task-Oriented Dialogue

Pre-training for end-to-end task-oriented dialogue systems (EToDs) is a challenging task due to its unique knowledge base query (accuracy) need and lack of sufficient training data (fluency). In this paper, we try to mitigate the above challenges by introducing a modularized pre-training framework for EToDs, which achieves to effectively improve both accuracy and fluency of EToDs through a pre-training paradigm. The core insight is a modular design by decomposing EToDs into a generation (fluency) module and a knowledge-retriever (accuracy) module, which allows us to optimize each module by pre-training these two sub-modules with different well-designed pre-training tasks, respectively. In addition, such a modularized paradigm enables us to make full use of large amounts of KB-free dialogue corpus for the pre-training generation module, which can alleviate the insufficient training problem. Furthermore, we introduce a new consistency-guided data augmentation (CGDA) strategy to cope with the data scarcity problem to better pre-train the knowledge-retriever module. Finally, we fine-tune the pre-trained generation module and knowledge-retriever module jointly. Experimental results on three datasets show that our model achieve superior performance in terms of both fluency and accuracy. To our knowledge, this is the first work to explore modularized pre-training methods for EToDs.

I. Introduction

Task-oriented dialogue systems (ToDs) can complete user goals such as hotel bookings and restaurant reservations, which gains increasing attention. Traditional ToDs consists of modularly connected components for natural language understanding (NLU) [1], dialogue state tracking (DST) [2], dialogue policy (DP) [3] and natural language generation (NLG) [4] module. In recent years, end-to-end task-oriented dialogue systems (EToDs) has emerged in the literature, which use a unified sequence-to-sequence model to generate a response given a dialogue history and knowledge base (KB) [5]. For example, given the dialogue history “Send me to the nearest gas station, I want to fuel my car.” in the first turn and the corresponding knowledge base in Fig. 1, EToDs can directly produce the system response “The nearest gas station is valero at 200 Alester Ave, 7 miles away setting directions now”, which does not need any intermediate supervision.