Bayesian Optimization-Based Global Optimal Rank Selection for Compression of Convolutional Neural Networks

Recently, convolutional neural network (CNN) compression via low-rank decomposition has achieved remarkable performance. Finding the optimal rank is a crucial problem because rank is the only hyperparameter for controlling computational complexity and accuracy in compressed CNNs. In this paper, we propose a global optimal rank selection method based on Bayesian optimization (BayesOpt), which is a machine learning based global optimization technique. By utilizing both a simple objective function and a proper optimization scheme, the proposed method produces a global optimal rank that provides a good trade-off between computational complexity and accuracy degradation. In addition, our method also reflects the correlation of each rank in multi-rank selection, and is able to flexibly yield an optimal rank with a given fixed compression ratio. Experimental results indicate that the proposed algorithm can identify the global optimal rank regardless of the huge size of dataset or the various structural features of CNNs. In all experiments on multi-rank selection, the proposed method produces the rank with higher accuracy and lower computational complexity than the state-of-the-art rank selection method, variational Bayesian matrix factorization (VBMF).