Effective Masked Language Modeling for Temporal Commonsense Reasoning

We propose an enhanced language model for temporal commonsense inference. Through a specially designed masked language modeling task, we equip our model with temporal commonsense knowledge. Specifically, we propose different masking strategies for temporal commonsense reasoning. We evaluate our model on a challenging temporal commonsense reasoning dataset. Our experiments show that the method that takes into account the structure of the target dataset, in addition to time-related features, improved the accuracy by approximately 4.8% compared to the standard fine-tuning approach. This confirms that tailoring a pre-trained language model to the domain of a target task is effective.