Dynamic Structured Pruning With Novel Filter Importance and Leaky Masking Based on Convolution and Batch Normalization Parameters

Various pruning methods have been proposed to solve the overparameterized problem in deep neural networks. Most of the structured pruning methods have used magnitude-based filter importance to remove unnecessary filters. Usually, Convolutional Neural Networks (CNN) consist of blocks that proceed with batch normalization (BN) operations after convolutional (Conv) operations. Each element is calculated through cooperation between Conv and BN, so both Conv and BN parameters must be considered together in pruning. However, previous pruning methods independently used the norm of parameters, either Conv weights or BN scales, as filter importance, ignoring these CNN structures. With this intuition, we propose a new magnitude-based filter importance method that considers both Conv weights and BN scales, and provide evidence of importance through experimental analysis on the rank of the feature map as well as mathematical analysis on the feature distortion. Furthermore, in recent works, dynamically applying a mask to recover weights has enabled more accurate pruning. However, when the weights are multiplied by zero mask values, the gradients become zero, which prevents updating the weights. We name this problem the zero gradient transferring. To solve this problem, we propose a leaky masking method that replaces the zero value of the mask with a positive constant. As a result, we solve the zero gradient transferring through the leaky masking method, enabling more accurate dynamic structured pruning. We denote our proposed method as CoBaL that incorporates Conv and BN based filter importance and Leaky mask into dynamic structured pruning. Experimental results show that our CoBaL compresses 50.09% parameters and 32.51% FLOPs on ResNet 56 with the CIFAR-10 with slightly improved accuracy. Also, we possess comparable results on ImageNet with 50.74% and 29.38% reduction in the number of parameters and FLOPs at ResNet18, respectively.