nDirect2: A High-Performance Library for Direct Convolutions on Multicore CPUs

Convolution kernels are widely seen in high-performance computing (HPC) and deep learning (DL) workloads and are often responsible for performance bottlenecks. Prior works have demonstrated that the direct convolution approach can outperform the conventional convolution implementation. Although well-studied, the existing approaches for direct convolution are either incompatible with the mainstream DL data layouts or lead to suboptimal performance. We design nDirect2, a novel direct convolution approach that targets multi-core CPUs commonly found in smartphones and HPC systems. nDirect2 is compatible with the data layout formats used by mainstream DL frameworks and offers new optimizations for the computational kernel, data packing, advanced operator fusion, and parallelization. We evaluate nDirect2 by applying it to representative convolution kernels and demonstrating how well it performs on four distinct ARM-based CPUs and an X86-based CPU. Experimental results show that nDirect2 outperforms four state-of-the-art convolution approaches across most evaluation cases and hardware architectures.

I. Introduction

Despite the recent success of Transformer-based neural networks, convolutional-based neural networks (CNNs) remain widely used for image classification [1], object detection [2], and natural language processing [3]. At the heart of CNNs is the convolution operation (Conv) [4], which often represents more than 90% of the execution time in classical CNN models like VGG [5]. As such, considerable interest has been put in optimizing convolution implementations to accelerate CNNs [6], [7].