A Soft-Hard Collaborative CNN Inference Acceleration System Based on NP Cores of DPU

With the continuous expansion of network scale, data centers are increasingly adopting Dedicated Processing Units (DPU) to offload network tasks. However, AI deep learning tasks heavily burden the host, and such computation-intensive tasks struggle on the DPU due to limited storage and constrained computing capabilities. To address this issue, Our research focuses on the NP multi-cores architecture of DPU and innovatively leverages these micro-cores, originally intended for network packet forwarding, to deploy AI neural network inference on the DPU, achieving the first full offloading of convolutional neural network inference tasks to the NP cores of the DPU. We propose an AI inference acceleration system based on soft-hard collaboration, which innovatively proposes model format self-adjustment, block sliding memory access algorithm, and multi-core parallel pipeline scheduling strategy by combining hardware characteristics. Based on these methods, we have successfully implemented a complete CNN inference process for network anomaly traffic detection and ransomware detection scenarios on Huawei DPU chips. The system can perform CNN inference on NP cores with only 2kB of running memory, effectively reducing the peak runtime memory by up to 70%, and flexibly scheduling through the deployment of integer computation paths and soft floating-point computation paths. Compared to the direct offloading method, this system achieves 17–22x acceleration on the NP cores of the DPU, even if the performance of the NP cores is only one tenth of that of the CPU in the host, it can still achieve inference latency comparable to that of the host.