LPC: Efficient Lossless Parameter Compression for Deploying LLM Inference on Edge Systems

The inference latency of large language models (LLMs) on edge systems is often bottlenecked by the limited memory bandwidth between host and accelerator, primarily due to repeated parameter transfers necessitated by the accelerator’s restricted memory capacity. In contrast to lossy model compression methods that address this issue at the expense of model accuracy and time-consuming, model-specific tuning, lossless compression retains the accuracy and offers plug-andplay usability, making it an appealing alternative. However, its practical deployment at runtime is hindered by the inherent tradeoffs among compression ratio, decompression throughput, and hardware overhead. To tackle these challenges, we present a Lossless Parameter Compression (LPC) technique that exploits the intrinsic numerical characteristics of LLM parameters. Through a combination of lightweight value remapping scheme and the hardware-friendly LZ4 compression algorithm, LPC achieves same compression ratio as more sophisticated entropycoding algorithms (e.g., Deflate), while offering same decompression bandwidth at 4.5× lower area cost. We validated the benefit of LPC to inference performance in a real host-accelerator system environment, employing an open-source CNN accelerator with its hardware and software stack redesigned to accommodate PyTorch-based LLMs. Evaluation across a broad spectrum of models demonstrated 29.1–29.4% and 19.4–43.4% reductions in parameter offloading traffic for BF16 and INT8 formats, leading to inference speedups of up to 1.42× and 1.76×, respectively.