Composition of Experts on the SN40L Reconfigurable Dataflow Unit

Monolithic large language models (LLMs) pose significant challenges in training and serving during an active deployment. By contrast, composition of experts (CoE) is a modular approach that lowers the cost and complexity of training and serving. In this article, we explore unique hardware challenges for CoE models, such as lower operational intensity and the cost of switching between models. We describe the SambaNova SN40L Reconfigurable Dataflow Unit (RDU), which combines streaming dataflow and a new three-tier memory system with static random-access memory (RAM), high-bandwidth memory, and double data rate dynamic RAM. A single eight-socket SN40L node achieves speedups between 2 and 13× due to aggressive operator fusion over an optimized baseline. The SN40L node deploys Samba-CoE, a 1 trillion-parameter CoE with a 19× smaller machine footprint, speeds up model switching time by 15–31× and achieves an overall speedup of 3.7× over a DGX H100 and 6.6× over a DGX A100.