Optimizing Fully Spectral Convolutional Neural Networks on FPGA

Computing convolutional neural networks (CNNs) in frequency domain largely reduces the number of operations for training and inference of CNNs. However, existing designs with such an idea require repeated spatial- and frequency-domain switching due to the absence of nonlinear functions in the frequency domain, as such it makes the benefit less attractive for low-latency inference. This paper presents a fully spectral CNN approach by proposing a novel spectral Rectified Linear Unit (ReLU) activation function to avoid multiple compute-intensive domain transformations. Our proposal maintains the non-linearity in the network and takes into account the hardware efficiency in algorithm design. Then, a customized hardware architecture is proposed to implement the designed spectral network on FPGA device. Optimization techniques such as merging layers and DSP configuration are performed to further increase the inference speed. The hardware accelerator is implemented on Intel's Arria 10 device. Experimental results indicate that the proposed spectral-based design achieves 6.0× and 4.2× speedup compared to the spatial and FFT-based designs on the same device respectively, within only 0.5 percent accuracy loss on MNIST dataset.