Quantization Strategy for Pareto-optimally Low-cost and Accurate CNN

Quantization is an effective technique to reduce memory and computational costs for inference of convolutional neural networks (CNNs). However, it has not been clarified which model can achieve higher recognition accuracy with lower memory and computational costs: a fat model (large number of parameters) quantized to an extremely low bit width (e.g., 1 or 2 bits) or a slim model (small number of parameters) quantized to moderately low bit width (e.g., 4 or 5 bits). To answer this question, we define a metric that combines the number of parameters and computations with bit widths of quantized weight parameters. Using this metric, we demonstrate that Pareto-optimal performance, where the best accuracy is obtained at a given memory or computational cost, is achieved when a slim model is moderately quantized rather than when a fat model is extremely quantized. Moreover, employing a strategy based on this finding, we empirically show that the Pareto frontier is improved by 4.3× under a post-training quantization scenario on the ImageNet dataset.