23.5 MAE: A 3nm 0.168mm2 576MAC Mini AutoEncoder with Line-Based Depth-First Scheduling for Generative AI in Vision on Edge Devices

Generative AI for vision has demonstrated significant potential to revolutionize user experiences through its capability to generate images with exceptional perceptual quality. The ability of generative models to handle diverse input modalities enables various image synthesis and editing applications, such as text-to-image, high-resolution image restoration, and image inpainting. Supporting these AI applications on edge devices, however, requires solutions that ensure low latency while minimizing bandwidth, power, and area at the same time. For these needs, a heterogeneous multi-core system can leverage a general neural processing unit (NPU) for high throughput and low latency, and use a specialized tiny NPU for low-power scenarios. In this system, each task is allocated to a more efficient core to exploit complementary strengths of each core. For instance, when deploying the most promising and popular generative diffusion models [1], the two cores can operate collaboratively: the specialized tiny NPU handles the autoencoder with high utilization and low power, while the general NPU executes the latency-dominant denoising U-Net and conditioning modules to minimize latency. However, compared to the general NPU, designing the specialized tiny NPU must conform to a very limited area budget, which inherently restricts the number of MAC units and the amount of on-chip memory. This presents two challenges: 1) the reduced on-chip memory necessitates more energy-consuming DRAM access. In a layer-by-layer schedule, the energy consumed by DRAM access accounts for 40-to-80% of the total energy, critically affecting the low-power objective; 2) temporal under-utilization can occur when calculations are limited by IO bandwidth or non-MAC operations, limiting overall performance. In this paper, we present MAE, a domain-specific tiny NPU fabricated in 3nm technology, which efficiently runs the tiny autoencoder for stable diffusion (TAESD) [2]–[6]. The chip features: 1) line-based depth-first fusion with branch-handling to reduce external memory access (EMA), 2) a local H-reuse caching and direct link mechanism for improving temporal utilization, and 3) a scheduling optimization framework to globally minimize latency and EMA. These innovations enable MAE to integrate 576 MACs within a 0.168mm² area, achieving an energy efficiency of 12.38TOPS/W and 84.5% utilization on the target network.