Accelerate Mini-batch Machine Learning Training With Dynamic Batch Size Fitting

Mini-batch Stochastic Gradient Descent (MGD) is one of the most widely used methods in Machine Learning (ML) model training. Typically, before a training process starts, researchers should manually set a fixed batch size, which is a hyper-parameter indicating the size of the random slice of the whole dataset that is trained in a single iteration. In this paper, we propose a light-weight dynamic batch size fitting algorithm based on online efficient evaluation, which has the ability of automatically tuning batch size during the train process to reach a best-so-far efficiency, but with little overhead. The experimental results have demonstrated that the algorithm is more effective compared with the commonly used fixed settings.