Detection of Day-Based Health Evidence with Pretrained Large Language Models: A Case of COVID-19 Symptoms in Social Media Posts

Gathering the information pertaining to health evidence occurring on particular days can help understand the progression of a disease over time, and such evidence as symptoms was widely shared by the social media users during the COVID-19 pandemic. Identifying this type of evidence is challenging. In this work, we investigated pretrained large language models on their ability to identify the day-based COVID-19 symptom mentions in Twitter posts. Our results on a corpus of 635 tweets show that without any supervision and optimization both GPT-3.5 and GPT 4 models achieved impressive performance, much better than Web-based ChatGPT and Google Bard. In addition, we explored and utilized GPT-4’s ability to determine the number of matches between a list of predicted symptom expressions and a list of annotated symptom expressions, reducing the effort of designing a sophisticated algorithm for finding the matches.