Determining the Logical Relation between Two Sentences by Using the Masked Language Model of BERT

In this paper, we evaluated the suitability of BERT^,s Masked Language Model to determine the relation between two sentences, s1 and s2. A mask word was placed immediately before s2, creating a BERT input sequence of the form "[CLS] s1 [SEP] [MASK] s2 [SEP]." The relation between the sentences was determined from the most likely conjunctives for the [MASK] token, as predicted by BERT. In this study, we used a list of most common conjunctives and a set of sentence pairs containing those conjunctives to test whether the conjunction could be predicted when it is masked. We compared our results with the case of solving the sentence relation problem by using two different supervised learning methods.