Network Pruning Based On Architecture Search and Intermediate Representation

Network pruning is widely used for compressing large neural networks to save computational resources. In traditional pruning methods, predefined hyperparameters are often required to determine the network structure of the target small network. However, too many hyperparameters are often undesirable. Therefore, we use the transformable architecture search (TAS) method to dynamically search the network structure of each layer when pruning the network width. In the TAS method, the channels number of the pruned network in each layer is represented by a learnable probability distribution. By minimizing computation cost, the probability distribution can be calculated and used to get the width configuration of the target pruned network. Then, the depth of the network was compressed based on the model get in the previous step. The method for compressing depth is block-wise intermediate representation training. This method is based on the hint training, where the network depth is compressed by comparing the intermediate representation of each layer of two equally wide teacher and student models. In the experiments, about 0.4% improvement over the existing method was viewed for the ResNet network on both CIFAR10 and CIFAR100 datasets.