Simultaneous Learning and Compression for Convolution Neural Networks

Neural network compression techniques almost always operate on pretrained filters. In this paper we propose a sparse training method for simultaneous compression and learning, which operates in the eigen space of the randomly initialized filters and learns to compactly represent the network as it trains from scratch. This eliminates the usual two-step process of having to first train the network, and then compressing it afterwards. To learn the sparse representations we enforce group L1 regularization on the linear combination weights of eigen filters. This results in the recombined filters which have low rank and can be readily compressed with standard pruning and low rank approximation methods. Moreover we show that the L1 norm of the linear combination weights can be used as a proxy for the filter importance for pruning. We demonstrate the effectiveness of our method by applying it to several CNN architectures, and show that our method directly achieves the best compression with competitive performance accuracy as compared to state of the art methods for compressing pre-trained networks.

1. INTRODUCTION

Two popular neural network compression methods are filter pruning and low rank approximation. Pruning typically starts with a filter importance score estimation, where a dedicated algorithm estimates the importance score of each pretrained filter in the model. There are several ways to estimate filter importance, some methods (i.e. [1], [2], [3]) estimate the contribution of each filter to some loss and use it as a proxy for importance while others estimate filter importance based on the statistics of the output feature map [4]. Least important filters are then removed from the model, either iteratively or all at once and the resulting compressed model then is finetuned to recover any loss of performance.