Abstract:
Quantization is a promising technique to reduce the computation and storage costs of DNNs. Low-bit ( \leq8 bits) precision training remains an open problem due to the ...Show MoreMetadata
Abstract:
Quantization is a promising technique to reduce the computation and storage costs of DNNs. Low-bit ( \leq8 bits) precision training remains an open problem due to the difficulty of gradient quantization. In this paper, we find two long-standing misunderstandings of the bias of gradient quantization noise. First, the large bias of gradient quantization noise, instead of the variance, is the key factor of training accuracy loss. Second, the widely used stochastic rounding cannot solve the training crash problem caused by the gradient quantization bias in practice. Moreover, we find that the asymmetric distribution of gradients causes a large bias of gradient quantization noise. Based on our findings, we propose a novel adaptive piecewise quantization method to effectively limit the bias of gradient quantization noise. Accordingly, we propose a new data format, Piecewise Fixed Point (PWF), to present data after quantization. We apply our method to different applications including image classification, machine translation, optical character recognition, and text classification. We achieve approximately 1.9\sim 3.5\times speedup compared with full precision training with an accuracy loss of less than 0.5%. To the best of our knowledge, this is the first work to quantize gradients of all layers to 8 bits in both large-scale CNN and RNN training with negligible accuracy loss.
Published in: IEEE Transactions on Image Processing ( Volume: 31)
Funding Agency:
Research on Image Classification Improvement Based on Convolutional Neural Networks with Mixed Training
Yongyue Zhang,Junhao Zhang,Wenhao Zhou
Image classification of fine-grained fashion image based on style using pre-trained convolutional neural network
Yian Seo,Kyung-shik Shin
Iterative Random Training Sampling Convolutional Neural Network for Hyperspectral Image Classification
Chein-I Chang,Chia-Chen Liang,Peter Fuming Hu
SAR Image Classification Using Mixed Spatial-Spectral Information and Pre-trained Convolutional Neural Networks
Melisa Unsalan,Anamaria Radoi,Mihai Datcu
X-Ray Chest Image Classification by A Small-Sized Convolutional Neural Network
Ege Kesim,Zumray Dokur,Tamer Olmez
Evaluation of Convolutional Neural Network Architectures for Chart Image Classification
Paulo Chagas,Rafael Akiyama,Aruanda Meiguins,Carlos Santos,Filipe Saraiva,Bianchi Meiguins,Jefferson Morais
Fully Convolutional Neural Network Structure and Its Loss Function for Image Classification
Qiuyu Zhu,Xuewen Zu
Accelerating the Training Process of Convolutional Neural Networks for Image Classification by Dropping Training Samples Out
Naisen Yang,Hong Tang,Jianwei Yue,Xin Yang,Zhihua Xu
FloatSD: A New Weight Representation and Associated Update Method for Efficient Convolutional Neural Network Training
Po-Chen Lin,Mu-Kai Sun,Chuking Kung,Tzi-Dar Chiueh
Convolutional Neural Network (CNN) for Image Classification of Indonesia Sign Language Using Tensorflow
Olivia Kembuan,Gladly Caren Rorimpandey,Soenandar Milian Tompunu Tengker