Optimizing Communication and Performance in Federated Learning for Large Language Models

LLMs(Large Language Models) has shown excellent performance in natural language processing and generation tasks. However, most LLMs learn by aggregating large amounts of data to a central server in a centralized environment, which causes problems such as data privacy infringement risk, high computing resource requirements, and increased operating costs. To solve these limitations, federated learning, which enables local learning without transmitting sensitive data to a central server, is attracting attention. In this study, we propose a new federated learning-based LLM Fine-Tuning strategy using the FlowerTune framework. The proposed methodology maximizes communication efficiency while ensuring data privacy in a distributed environment such as the medical domain, and verifies its performance based on pretrained LLM. This study conducted experiments in 10-round and 100-round settings using three medical datasets: PubMedQA, MedMCQA, and MedQA, and its main contributions are as follows. It demonstrated high performance by achieving an average accuracy of 63.26% even in limited communication rounds (10 rounds). We applied a LoRA-based parameter-efficient fine-tuning technique to maximize learning efficiency and solve the resource constraint problem between clients. The empirical findings indicate that the advocated approach effectively addresses the constraints associated with centralized learning methodologies, thereby attaining superior performance while simultaneously minimizing communication overhead. This investigation posits the feasibility of integrating federated learning and LLMs within contexts where data confidentiality is paramount, such as in the medical field, and establishes a groundwork for subsequent inquiries into effective federated learning strategies.