System-Technology Co-Optimization Methodology for LLM Accelerators with Advanced Packaging

Recent progress in large language models (LLMs) suggests the feasibility of their deployment on personal devices with model size reduction to a few to dozens of GB. Still, intermediate data’s computing needs are intensive, requiring frequent data reloading from the high-bandwidth memory (HBM). Today’s HBM bandwidth is limited by the number of channels embedded in a 2.5D integrated system. Advanced packaging techniques such as through silicon vias (TSV) and Cu-Cu hybrid bonding (HB) could potentially provide higher bandwidth interconnects between memory and logic dies in a 3D integrated system, where the vertical interconnect can reduce the distance between memory and logic, reducing the total energy consumption. However, this creates a large design exploration space for mixing and matching different packaging techniques and can result in complex thermal management issues due to the proximity of various components. In this work, we describe an evaluation methodology which is used to construct a framework capable of benchmarking system-level power, performance, and area (PPA) metrics for 2.5D/3D integrated systems for LLM accelerators. Additionally, we utilize the framework to conduct a detailed analysis to identify the bottlenecks for training and inference across various models and batch sizes. It is observed that the memory bandwidth and routing energy bottlenecks the inference performance, and the available compute bottlenecks the training performance. Finally, we perform thermal evaluations to observe the trade-off between peak operating temperature and the throughput across different packaging configurations.