Sparse CNN Architecture Search (Scas)

Advent of deep neural networks has revolutionized Computer Vision. However, designing of such models with high accuracy and low computation requirements is a difficult task and needs extensive human expertise. Recent advances in Neural Architecture Search use various methods like Deep Reinforcement Learning, Evolutionary methods, Gradient Descent, HyperNetworks etc. to automatically generate neural networks with high level of accuracy. However, large size of such generated models limit their practical use. Recent findings about lottery ticket hypothesis suggest the existence of sparse subnetworks (winning tickets) which can reach the accuracy comparable to that of original dense network. In this paper, we present a method for leveraging redundancies inherent to deep Convolutional Neural Networks (CNN) to guide the generation of sparse CNN models (to find the architectures with winning tickets) without significant loss in accuracy. We evaluate our proposed method with different NAS methods on CFAR-10,CIFAR-100 and MNIST datasets. Our results show a reduction ranging from $2\times \mathrm{to} 12\times$ in terms of model size and $2\times \mathrm{to}19\times$ in terms of number of MAC operations with less than 1% drop in accuracy.