Efficient ConvNet Optimization through Width Modification Guided with Gaussian Processes

Computer vision has been greatly advanced with Convolutional Neural Networks (CNN or ConvNet), which generally have high demand on computational resources during training and/or inference. In this work, we focus on techniques to reduce resource needs of a CNN without largely affecting its accuracy. In particular, for the widely utilized ResNets that contain ConvNet layers organized in groups, we propose the Structured Coarse Block Rescaling (SCBR) to remove whole CNN channels from a group of ConvNet layers by adjusting their width together to circumvent the prohibitive costs in training of fine-grained model compression approaches. For a set of selected width modifiers, we experimentally evaluated the SCBR-pruned ResNets with the CIFAR dataset where the model size can be reduced to below 20% with less than 2% accuracy reduction when comparing to their baseline models. Furthermore, an efficient Gaussian Process (GP) guided search scheme is proposed to systematically explore impacts of SCBR across all width modifiers. Based on the derived ResNets from the set of selected width modifiers, the resulting GP model can predict performance of a SCBR-pruned ResNet with any width modifier, which has been experimentally verified with accuracy errors mostly below 0.5%.