Design Considerations for LLM Inference in Data Centers: Chip and Interconnect

This paper presents a cost-efficient chip prototype optimized for large language model (LLM) inference. We identify four key specifications – computational FLOPs (flops), memory bandwidth (bw), memory capacity (cap), and interconnect latency –and introduce two ratios (bw/cap and flops/ bw) to guide chip design. Through an analysis of LLM inference workflows, we show that memory bandwidth is critical for LLM decoding, while memory capacity and computational FLOPs are less important than in training. Based on these insights, we systematically propose a design with reduced memory capacity, increased bandwidth, and an interconnect topology design principle optimized for both interconnect latency and bandwidth.