Simultaneous Learning and Compression for Convolution Neural Networks

Neural network compression techniques almost always operate on pretrained filters. In this paper we propose a sparse training method for simultaneous compression and learning, which operates in the eigen space of the randomly initialized filters and learns to compactly represent the network as it trains from scratch. This eliminates the usual two-step process of having to first train the network, and then compressing it afterwards. To learn the sparse representations we enforce group L1 regularization on the linear combination weights of eigen filters. This results in the recombined filters which have low rank and can be readily compressed with standard pruning and low rank approximation methods. Moreover we show that the L1 norm of the linear combination weights can be used as a proxy for the filter importance for pruning. We demonstrate the effectiveness of our method by applying it to several CNN architectures, and show that our method directly achieves the best compression with competitive performance accuracy as compared to state of the art methods for compressing pre-trained networks.