MedSeq2Seq: A Medical Knowledge Enriched Sequence to Sequence Learning Model for COVID-19 Diagnosis

The COVID-19 pandemic has had a severe impact on humans’ lives and and healthcare systems worldwide. How to early, fastly and accurately diagnose infected patients via multimodal learning is now a research focus. The central challenges in this task mainly lie on multi-modal data representation and multi-modal feature fusion. To solve such challenges, we propose a medical knowledge enriched multi-modal sequence to sequence learning model, termed MedSeq2Seq. The key components include two attention mechanisms, viz. intra-modal (I
a
) and inter-model (I
e
) attentions, and a medical knowledge augmentation mechanism. The former two mechanisms are to learn multi-modal refined representation, while the latter aims to incorporate external medical knowledge into the proposed model. The experimental results show the effectiveness of the proposed MedSeq2Seq framework over state-of-the-art baselines with a significant improvement of 1%-2%.