Configurable CNN Accelerator in Speech Processing based on Vector Convolution

In speech applications, both input feature maps (IFMs) and kernels of neural networks are greatly diverse in shapes and sizes, which poses significant challenges to hardware acceleration. In this paper, a configurable CNN accelerator is introduced to make a good balance between the flexibility and efficiency for various neural network models in speech processing. The vector convolution scheme is first proposed by re-arrangement of IFM rows and weight values in vectors, by which the element convolution is converted into vector operations to break the limit of kernel-centric processing. The structure of vector processing element (VPE) is introduced to fit the continuous scaling down of IFMs with little control overheads, and the architecture of the CNN accelerator is proposed accordingly. FPGA implementation results demonstrate that the throughput is increased by 86% by the proposed architecture compared to state-of-the-art FPGA accelerators for the VGG16 network, while high DSP utilization is guaranteed for both 1D and 2D CNNs with various input sizes.