Implicit Discourse Relation Recognition by Scoring Special Tokens

Implicit discourse relation recognition is one of the most difficult tasks in natural language understanding. Because of the lack of explicit connectives, the ability to extract logical information between the two discourse arguments is highly required. Previous studies mainly focus on designing complex neural network layers and various kinds of interactions between the two arguments, without further exploring the logical semantics in the pre-trained language models. We propose a novel method that utilizes the power of the pre-trained language model (RoBERTa) by introducing different kinds of extra special tokens, which can represent the relations between the discourse arguments respectively. On one hand, these special tokens learn to aggregate category features that profit classification; on the other hand, they can also be regarded as new “words” that the prediction of them inspires the pre-train language model. To effectively learn these special tokens, a scorer is then trained to give the discourse connected by corresponding special tokens higher scores than those with other special tokens. The experiments show the result of our approach exceeds our baseline by 4.24% F1, and the state-of-the-art model by approximately 2.37% F1 on the four-way classification of the PDTB 2.0 dataset.