Exploring Group Sparsity using Dynamic Sparse Training

Group-level pruning is a model compression method that can accelerate models on general hardware while maintaining the accuracy of models even at high sparsity. Existing group-level pruning studies have a limitation in that pruning is performed by evaluating the importance of the weight group based on a single global threshold, and thus the accuracy loss is significant. In this paper, we propose Grouped DST, which applies Dynamic Sparse Training to group-level pruning to learn the criteria for each filter to determine the importance of weight groups. We validate our technique at ResNet-20 on CIFAR-10. The model compressed with Grouped DST achieve 3.77% a lower accuracy loss compared to that pruned with a global pruning threshold at 95% of sparsity.