Two-Step Masked Language Model for Domain-Adapting Multi-Modal Task-Oriented Dialogue Systems

The next generation of artificial intelligence (AI) is required to be capable of proper communication to enable eloquent interaction with human beings. Thus, AI models require powerful language understanding and generation capabilities. Therefore, designing an adequate dialogue-specific task is an important topic in dialogue research. In this article, we improve our model that won in two out of four subtasks in SIMMC2.0 challenge using a dialogue-specific pre-training task, which learns the distinctive features of dialogues. In addition, we introduce an efficient method to pre-train the encoder-decoder transformer using an auxiliary task for the encoder. Experimental results indicate that superior performance is achieved compared to our previous model, revealing a new approach to improve task-oriented dialogue systems.