NPU Design with Optimized Hardware Utilization Through Layer Fusion Schedule in Model Compiler

Edge applications often demand efficient and optimized processing due to limited computational resources, particularly when deploying quantized models for real-time inference. To address these challenges, reducing memory transactions and optimizing arithmetic intensity becomes crucial for achieving better performance. Limited memory bandwidth significantly restricts data movement, leading to bottlenecks that hinder computational efficiency. In this paper, we propose a software-hardware co-design methodology to reduce memory access and enhance arithmetic intensity. Our approach involves leveraging a model compiler to automate operation remapping, layer-fusion, tiling, code generation, and instruction generation. Our compiler fuses planar layers (e.g., depthwise convolution, average pooling) and element-wise layers (e.g., residual addition) with convolution, pipelines the convolution in compute unit, a multiplier-accumulator (MAC) array, and planar operations in a post-process unit, a vector processing unit. It can maximize on-chip data reuse and minimize memory transactions. In this way, we achieve enhancing the hardware's computational capabilities for these operations and shortening latency. This integrated hardware-software strategy enables highly efficient execution of fused layers, resulting in substantial performance gains. Our results demonstrate that layer fusion through the compiler and pipelining these operations on the neural processing unit (NPU) hardware brings 44.4% reduction of memory transactions and 95.6% performance improvements by adding an extra post-processing unit for planar operations. These results show that co-designing NPU hardware architecture with a model compiler is crucial to developing edge AI accelerators.