When the Edge Meets Transformers: Distributed Inference with Transformer Models

Transformer models achieved significant break-throughs in a wide variety of applications, yet their exorbitant computation costs pose significant challenges when it comes to deploying these models for inference, especially on resource-constrained edge devices. In this paper, we introduce the concept of cross-device distributed inference to transformer models, which accelerates the speed of inference by distributing its workload among multiple edge devices. Unlike previous work designed for multi-GPU environments, the challenge of distributing inference workload on edge devices includes not only limited computation power, but also low bandwidth connections to exchange intermediate results. To address these challenges, we propose Voltage, a distributed inference system tailored for edge devices. By exploiting the inherent parallelizability of the input sequence, Voltage partitions the transformer inference workload based on positions to accelerate the inference speed. We also analyze the relationship between the partition settings and the computation complexity, which allows Voltage to adaptively select the most efficient computation scheme. To demonstrate its effectiveness and generalizability, the performance of Voltage has been evaluated in the context of well-known transformer models, and in a variety of experimental settings. Our results show that Voltage significantly outperforms tensor parallelism by reducing the communication size by 4 x, thereby accelerate the inference speed by up to 32.2 % compared with single device deployment.