DL-BERT: a time-aware double-level BERT-style model with pre-training for disease prediction

Disease prediction based on the Electronic Health Record (EHR) is an important task in healthcare. EHR records patients’ every visit by time, and there are many kinds of medical codes within a visit, therefore, EHR has characteristics of temporal irregularity and hierarchical structure. Some recent works employ BERT-style models to process EHR data for disease prediction. However, few of these models can give consideration to capture both the interaction between medical codes and the impact of temporal irregularity. To solve this problem, we propose the Double-Level BERT-style model (DL-BERT). Considering EHR’s hierarchical structure, the model contains a code-level and a visit-level representation layer which can learn the relationship between medical codes and temporal influence respectively. In the code-level representation layer, the model achieves the representation power by employing external medical ontologies to provide multi-resolution information of medical codes and the Transformer to embed medical codes. Besides, the model adopts two pre-training tasks to enhance the ability to capture the link between different kinds of codes. In the visit-level representation layer, DL-BERT utilizes a special time-aware Transformer to model temporal information. And the model adopts a visit-level pre-training task for better learning context information. Experiments are conducted on two real-world healthcare datasets and show that our model outperforms all baselines demonstrating the effectiveness of our model.