Comparative Study of Large Language Models for Lung-RADS Classification in Portuguese CT Reports

Lung cancer has the highest mortality rate among all cancer types for both males and females. It is estimated that lung cancer accounts for 21% of cancer deaths in each gender. This alarming statistic highlights the significant impact of lung cancer on overall cancer mortality, underscoring the urgent need for effective prevention, early detection, and treatment strategies to combat this disease. Lung cancer screening (LCS) is a process that involves carefully selecting high-risk individuals, primarily current or former heavy smokers. It includes annual low-dose computed tomography scans and meticulous interpretation of results followed by appropriate follow-up care. Adherence to LCS follow-up is essential for maximizing the life-saving benefits of this preventive measure. Multiple professional societies, such as the American College of Radiology (ACR) and Fleischner Society, have published guidelines for managing patients with pulmonary nodules. Lung CT Screening Reporting & Data System is a quality assurance tool designed to standardize the reporting of lung cancer screening CT scans and provide consistent management recommendations. In this context, this work aims to evaluate whether large language models (LLM) could accurately identify and extract lung nodules' characteristics from unstructured chest CT reports in the Portuguese, based on the Lung-RADS classification system. This work assessed the effectiveness of three LLMs: Gemini, GPT-4-o, Llama-3 70B, and a BERT model BioBERTpt. Our findings indicate that LLMs, especially GPT-4-o, have significant potential in automating the extraction of lung nodule characteristics for Lung-RADS classification, which could aid radiologists in their work. Notably, GPT-4-o with few-shot learning using Prompt 4 emerged as the best model, achieving an F1-score of 0.89. Our results highlight the potential of LLMs to assist radiologists in accurately classifying lung nodules according to the Lung-RADS criteria, streamlining the diagnostic process.