Deep Neural Network Pruning with Progressive Regularizer

Pruning is a pivotal approach in network compression. It not only encourages lightweight deep neural networks, but also helps to mitigate overfitting. Generally, regularization is used to guide more parameters towards zero and thus reduce the overall model complexity. Unfortunately, there are two issues remaining unsolved in regularization based pruning. One is that the optimal trade-off between regularization and loss minimization, often expressed via a scaling hyperparameter, needs to be found through extensive experimentation. The other one is the importance criteria that can reflect the true relative importance. The most widely used criterion is the magnitude-based, which has been argued to be inaccurate. To these two issues, in this paper, we propose a progressive regularization scheme, in which the factor scaling the regularization term is gradually increased during training, until the target sparsity for filter pruning is reached. Compared to the previous approach, the scaling factor is no longer a hyperparameter that needs to be tuned, but is replaced with a sparsity-aware parameter that increases progressively. In this way, the value of the scaling factor can be automatically aligned with the target sparsity, avoiding the drawbacks in its tuning. Furthermore, only parameters below a minimal and learnable soft threshold are pruned, therefore, informative parameters (even with small magnitudes) are optimally preserved. Consequently, the performance loss due to pruning is mitigated. Experiments with various models and benchmark datasets prove the effectiveness of the proposed method.