Low-Resource Efficient Multi-Stage Tuning Strategy for Biomedical Question Answering Task

The automated question-answering system plays a crucial role in improving the accuracy and efficiency of clinical decision-making. While large-scale language models perform prominently in the general domain, even surpassing human performance in certain aspects, challenges such as data privacy and the massive training costs affect the broader adoption of LLMs in biomedical domain. This study explores the strategies of efficient fine-tuning and optimization methods in biomedical domain. We introduce a multi-stage fine-tuning strategy that improves the accuracy of medical question-answering tasks significantly. Specifically, a contrastive learning technique based on multi-prompts is proposed, and a self-consistency voting approach is used to improve the accuracy of reasoning-required tasks. Experimental results on PubMedQA dataset reveal that even fine-tuning only 0.152% of the baseline’s parameters, our method still improves its performance, making it outperform the domain-specific pre-trained models and achieve performance comparable to GPT-4.