AsyncGrid: An Intra- and Inter-Layer Asynchronous Hybrid Parallelism System for Responsive Edge LLM Inference

Edge deployment of large language models (LLMs) is increasingly attractive due to its advantages in privacy, customization, and availability. However, edge environments face significant challenges in reducing Time-to-First-Token (TTFT). TTFT consists of (1) queuing delay and (2) prefill latency, both of which are exacerbated by edge‑resource constraints: the substantial computational demands of LLM inference grow superlinearly with prompt length, causing high prefill latency; and limited edge resources restrict prefill throughput, preventing the timely handling of incoming requests, thereby exacerbating queuing delays. Model parallelism is a commonly used solution in cloud-based systems, but directly applying it to edge environments proves ineffective. Intra-layer parallelism (e.g., tensor/sequence parallelism) can reduce prefill latency but suffers from frequent global synchronization, which bottlenecks prefill throughput due to edge-limited interconnection bandwidth. Inter-layer parallelism (e.g., pipeline parallelism) improves prefill throughput via fully asynchronous execution but retains high prefill latency due to stage-wise serialized computation. To address this dilemma, this paper leverages the properties of the causal attention mechanism in LLMs and proposes Intra-layer Asynchronous Parallelism (IAP), which performs intra-layer parallel computations to reduce prefill latency while avoiding global synchronization to mitigate prefill throughput bottlenecks. Moreover, considering communication sensitivity in intra-layer parallelism, this paper integrate IAP with inter-layer asynchronous parallelism into a unified plan space. This hybrid parallelism adapts to diverse hardware and request loads, enabling more effective TTFT optimization. To enable the end-to-end implementation of this hybrid parallelism, this paper propose AsyncGrid, an LLM inference system tailored for responsive edge LLM inference. AsyncGrid (1) models runtime overheads through a performance profiler, (2) employs an integer programming (IP) formulation to optimize execution plan, with the objective of minimizing latency while meeting throughput requirements, and (3) implements fine-grained communication optimization during runtime. A comprehensive evaluation on an edge testbed demonstrates AsyncGrid’s significant advantages over existing methods, achieving substantial improvements in both homogeneous and heterogeneous settings.