Approximation Algorithms and Simulated Annealing Heuristics for Row-and-Column Pruning of Deep Neural Networks

Convolutional neural networks (CNNs) have achieved immense success in computer vision and other field of science. Despite the achievements, state-of-the-art CNN models have grown to gigantic sizes that demand a lot of computing power and memory resources. As a model compression technique, parameter pruning can lower the computation and memory a CNN model needs. Previous work has proposed a parameter pruning method, which compresses the neural networks by eliminating the rows and columns of the weight matrices. Removing rows and columns preserves the dense structure of the weight matrix, rather than the sparse structure produced by unstructured pruning. Although row-and-column pruning effectively reduces the model size, making the model as compact as possible without sacrificing accuracy remains a challenge. We prove that the row-and-column pruning problem is an NP-complete problem, and we propose two approximation algorithms that provide solutions within twice the cost of the optimal solution. We also show that the approximation ratio cannot be improved for these two algorithms. Finally, we propose two schemes based on simulated annealing to solve the row-and-column pruning problem. The two proposed schemes improve accuracy by 2.7% and 1.6%, respectively, on the ResNet-18 model with 93% sparsity using the Food-101 dataset.