DH-PIM: Maximizing Computing Unit Utilization in Digital PIM by Dual Half Mode Extension

Transformer-based Large Language Models (LLMs) rely on both General Matrix-Matrix Multiplication (GEMM) and General Matrix-Vector Multiplication (GEMV) for inference. While existing Processing-in-Memory (PIM) architectures, like HBM-PIM and Newton, efficiently handle GEMV operations, they suffer from low computing unit utilization during GEMM due to frequent input preparation and activation cycles. To address this, we propose Dual Half-mode PIM (DH-PIM), an extension of the Half Division Mode from previous PipePIM. DH-PIM splits sense amplifiers, and computing units into halves, allowing them to execute different operations independently. Dual half mode, which allows both halves of the computing unit to process data broadcast from either half of the sense amplifier, significantly enhances utilization via overlapped and pipelined execution. Our simulations show DH-PIM improves GEMM performance by up to 2.06x in GEMM kernels, with about 1.46x speedup in LLM tasks, compared to HBM-PIM. These performance improvements are achieved with minimal architectural modifications, resulting in low area and energy overhead.