Libra: Novel LLM Token Streaming via Region-Based Task Scheduling and Token Bundling

The LLM serving systems are increasingly growing in popularity, as they provide various capabilities ranging from realtime translation to AI-driven chatbots. Recently, significant effort has been made to optimize server-side metrics such as token generation throughput, while the optimization of token streaming is simply overlooked, resulting in excessive network traffic and poor network utilization. In this paper, we introduce user regions to relieve the network issues, since users from the same region (e.g., universities and business zones) are likely to share similar behaviors to access LLM serving systems (e.g., they are active in a period of time). In this context, we propose Libra, a proxy-cloud collaborative serving system, where the cloud generates and bundles the tokens in terms of user regions and region-based proxy extracts and repacks the received token bundle to their corresponding users. At its core, we design an online region-based task scheduling algorithm with a provable performance to optimize user QoE and system overhead over time. Our evaluations show that Libra outperforms the state-of-the-art LLM serving systems (without user regions), such as vLLM, VTC and Andes, by up to $2.1 \times$ in the Time-Between-Tokens (TBT) metric and $78.4 \times$ in the number of packets. In addition, it achieves a 32.6 % reduction in TBT compared to other alternative algorithms (with user regions).