WDMoE: Wireless Distributed Large Language Models with Mixture of Experts

Large Language Models (LLMs) have achieved significant success in various natural language processing tasks, but how wireless networks can support LLMs has not been extensively studied. In this paper, we propose a wireless distributed LLMs paradigm based on Mixture of Experts (MoE), named WDMoE, through server-device collaboration at the wireless network edge. Specifically, we decompose the MoE layer in LLMs by deploying the gating network and the preceding neural network layer at the edge server of the base station (BS), while distributing the expert networks across the mobile devices. This arrangement leverages the parallel capabilities of expert networks on distributed devices. Moreover, to overcome the instability of wireless communications, we design an expert selection policy by taking into account both the performance of the model and the end-to-end latency, which includes both transmission delay and inference delay. Evaluations conducted across various LLMs and multiple datasets demonstrate that WDMoE not only outperforms existing models, such as Llama 2 with 70 billion parameters, but also significantly reduces end-to-end latency.