CNN Acceleration with Joint Optimization of Practical PIM and GPU on Embedded Devices

The lightweight AI models are proposed for embedded devices to mitigate computation and memory requirements. However, it still faces the memory wall problem (i.e., data movement between computing units and memory). Many processing-in-memory (PIM) based accelerators have been proposed to conquer this problem. Whereas, in practical deployment, there are still many challenges that need to be addressed.In this paper, we propose an embedded GPU-PIM integrated framework, with a fabricated HBM-PIM device from Samsung, to accelerate lightweight CNN inference. Considering various characteristics of different layers in lightweight CNNs, we first propose a fine-grained roofline model to guide the layer offloading to PIM cores. To fully utilize the computation and bandwidth resources of both GPU and PIM, we model the task allocation into a multi-dimensional knapsack problem (MDKP), which is addressed by a dynamic programming algorithm. Meanwhile, considering some practical issues such as PIM mode switch and memory address interleaving during the offloading, we further introduce a PIM-aware memory layout optimization strategy to reduce unnecessary inter-channel memory accesses and PIM mode switches. Experiment results show that our proposed framework achieves averagely the 5.9X and 37.9% inference performance boosts compared to the original GPU-only approach and the GPU-HBM approach, respectively.