Acceleration of Large Language Models with Emerging Compute and Communication Technology

Large Language Models (LLMs) are widely adopted for natural language processing (NLP) and other machine intelligence tasks due to their superior performance. However, their large parameter sizes and often densely connected computational structures lead to substantial computational and communication overheads. Traditional GPU-based platforms face challenges in scalability, fabrication cost, and energy efficiency for such workloads. To address these limitations, communication-and energy-efficient heterogeneous 2.5D chiplet-based with in-memory computing (IMC) and floating-point units (FPUs) can be used. The processing units are interconnected via a network-on-package (NoP) that combines wired and dynamically reconfigurable wireless links. Thorough experimental evaluation with a wide range of transformer algorithms shows that the 2.5D system demonstrates up to 1600× and 972× improvement in energy consumption and latency with respect to NVIDIA Orin Nano, and up to 39.32% and 38.86% in communication latency and energy respectively compared to existing 2.5D systems.