Improving Bi-Real Net with block-wise quantization and multiple-steps binarization on activation

Quantization neural networks use low-precision lowbit for both weights and activations, which are used for portable, low-power devices. And binary neural networks (BNNs) is an extreme case of quantization network in which both weights and activations are in a 1-bit representation. Binary neural networks replace expensive convolutional operations with fast bitwise operations such as xnor and popcount. While being efficient in the inference task, training BNN is very difficult because the training process is easy to be stuck on bad local minima. Besides, BNNs also suffer a problem of severe accuracy degradation. So how to train binary neural networks more effectively with higher accuracy? We answer this question by proposing a new training strategy on state-of-the-art binary neural network Bi-Real net. Our proposed strategy uses 3-steps binarization for activations, which helps to gradually reduce quantization error in the training phase. Also, we apply block-wise binarization to quantize block by block when training. Finally, we add a new regularization term into loss, to drive weights to -1 or 1. The comprehensive experiments on ImageNet show that our proposed training algorithm surpasses other binarization methods.