Robustly Pre-Trained Neural Model for Direct Temporal Relation Extraction

In this work, we studied methods to identify direct (intra-sentence) temporal relationships between clinical events and temporal expressions. In particular, we compared several variants of BERT, sampling strategies to balance training instances, and different model sizes. Our results show that the large RoBERTa model has improved overall performance by 0.0864 absolute F measure (on the 1.00 scale), thus reducing the error rate by 24% relative to the previous state-of-the-art performance achieved with SVM.