MedLoRA: Exploring LoRA and Quantization Combinations for Medical QA

Large Language Models (LLMs) have achieved remarkable progress across various NLP tasks. However, their adaptation to domain-specific applications such as medical question answering (MQA) faces two major bottlenecks: the cost of fine-tuning and the demand for efficient inference. To address these challenges, parameter-efficient tuning and model quantization have emerged as key strategies. In this paper, we propose MedLoRA, a comprehensive framework that systematically investigates the compatibility and performance of three representative LoRA methods (rsLoRA, DoRA, piSSA) with three popular quantization techniques (bitsandbytes, HQQ, EETQ) in MQA tasks. Beyond empirical benchmarking, we further explore the underlying mechanisms behind LoRAquantization interactions, such as DoRA’s gradient stability under low-bit inference. Through extensive experiments on four benchmark datasets-cMedQA-v1, cMedQA-v2, webMedQA, and MedQA-USMLE-we evaluate nine (LoRA $\times$ Quantization) combinations in terms of accuracy, parameter efficiency, and deployment feasibility. We also compare MedLoRA against fullparameter fine-tuning with quantization to contextualize performance-efficiency trade-offs. Our results reveal critical insights into how LoRA strategies interact with low-bit inference and highlight robust, lightweight configurations suitable for realworld medical applications.