Wafer Scale Interconnect and Pathfinding for Machine Learning Hardware (Invited)

The extreme compute requirements of Machine Learning (ML) drives an entirely new generation of hardware. The very compute-intensive ML training is generally done in data centers using re-purposed GPUs. This provides cost-efficient floating-point compute hardware that interfaces with the well-known TensorFlow ML platform. To run on a GPU, however, the training data and weights needs to be segmented to fit the limited on-chip memory and bandwidth. Cerebras takes a radically different approach with a massive
$22\mathrm{x}22\mathrm{cm}$
monolithic chip that contains over 400,000 powerful compute cores with 1.2 Trillion (with a T) transistors [1]. The massive Wafer Scale Engine (WSE) allows the entire ML model including all weights to remain stationary in hardware while only the training data is streamed in at very high speed. The result is a dramatic speedup of the training process. In this presentation we will address how the ML kernels are synthesized, placed and routed on the WSE compute fabric. In more detail we will address the unique interconnect properties of the compute fabric, and the EDA-inspired solutions to efficiently map the compute kernels.