Memory reduction method for deep neural network training

Training deep neural networks requires a large amount of memory, making very deep neural networks difficult to fit on accelerator memories. In order to overcome this limitation, we present a method to reduce the amount of memory for training a deep neural network. The method enables to suppress memory increase during the backward pass, by reusing the memory regions allocated for the forward pass. Experimental results exhibit our method reduced the occupied memory size in training by 44.7% on VGGNet with no accuracy affection. Our method also enabled training speedup by increasing the mini batch size up to double.