DisLLM: Distributed LLMs for Privacy Assurance in Resource-Constrained Environments

Large Language Models (LLMs) have revolutionized natural language processing, but deploying them in resource-constrained environments and privacy-sensitive domains remains challenging. This paper introduces the Distributed Large Language Model (DisLLM), a novel distributed learning framework that addresses privacy preservation and computational efficiency issues in LLM fine-tuning and inference. DisLLM leverages the Splitfed Learning (SFL) approach, combining Federated Learning (FL) and Split Learning (SL) benefits for privacy-preserving and computationally efficient LLM deployment. It splits the LLM into client-side and server-side components, allowing sensitive data to remain on client devices while distributing the computational load. An efficient fine-tuning technique called Low-Rank Adaptation (LoRA) is incorporated to reduce trainable parameters and memory footprint. Additionally, Local Differential Privacy (LDP) is used to add controlled noise to the sensitive data, further enhancing privacy preservation. Experimental evaluations on various text classification datasets demonstrate DisLLM’s effectiveness in model accuracy, convergence rate, and computational efficiency. Factors like the number of clients, cut layer position, and model variants are thoroughly investigated. Results show that DisLLM achieves comparable accuracy to centralized fine-tuning while preserving data privacy and optimizing resource utilization. Moreover, its ability to handle multiclass classification tasks with many classes emphasizes its robustness and versatility. Therefore, DisLLM represents a significant step forward in the practical deployment of LLMs, ensuring efficiency and privacy preservation.