Hardware-Software Co-design Approach for Deep Learning Inference

Hardware implementation of machine learning algorithms is a promising solution for higher performance and improved throughput. However, the most challenging task lies in the design of power, energy, and area efficient architectures that can be deployed in tightly constrained embedded systems. In this paper, we investigate the concept of intermediate exit branches in a CNN architecture aiming to reduce latency and energy. We propose a heterogeneous hardware-software based architecture for performing inference in feed-forward neural networks which is capable of early exiting through intermediate exits according to the difficulty level of the input. Our proposed architecture can be reconfigured to support different kernel sizes and is fully scalable. Our experiments show an increase in classification accuracy by 0.27% for MNIST dataset (LeNet) with 2.2× decrease in average computations and 1.07% for Cifar-10 dataset (AlexNet) with 1.55× decrease in average computations along with 1.39× reduction in energy consumption for LeNet architecture.