CovPTM : COVID-19 PreTrained Model Generic Model vs Biomedical Model Adaptation

Over the last couple of years, since the advent of the SARS-CoV-2 (COVID-19) pandemic, huge amounts of text data in the form of academic research, blogs and articles have been compiled in a bid to understand this disease and how to handle it. Diverse opinions from different experiments and theoretical understandings have created an overload of information. For every article arguing that COVID-19 is of natural origin, there is another arguing that it emerged from gain-of-function research in a laboratory. These contradicting views led to rapid widespread of misinformation. Therefore, we propose a domain specific Question-Answering system built on top of biomedical model that is capable of resolving abstruse scientific jargons to naive users. We also explored both generically trained models and biomedical domain adapted models for this work to comprehend the transfer performance towards target task. We observed that adapting a biomedical specific model is easier than adapting a generic model. This is because of the relatedness between the target and the source domain. From the results, we infer that domain adaptation of a biomedical language model to the COVID-QA dataset offers a significant improvement in performance and shows greatest improvement in the COVID-BioM-ALBERT model of more than 21% F1 score than the considered baseline.