Low Overhead PIM-to-PIM Communication on PCIe-based Multi-PIM Platforms for Executing Large-Scale AI Models

The emerging transformers suffer from the memory bottleneck because of their low data locality and large data sizes, and we are actively studying Processing in Memory (PIM), which computes inside memory, to overcome the bottleneck. However, as their model parameters get bigger, a single PIM device would be insufficient because of its limited memory capacity and computing resources. In this paper, we develop a low-overhead data communication method between PIMs on a PCIe-based multi-PIM platform. We adopt an XDMA-based PIM-to-PIM (P2P) direct data communication mechanism to eliminate redundant data movement between CPU and PIMs. Our P2P achieves 1.69x, 1.70x, 1.61x, and 1.63x speedups compared to the DMA using the system memory as a buffer when transferring 16MB, 32MB, 64MB, and 128MB data, respectively.