Asynchronous Variance Reduced SGD with Momentum Tuning in Edge Computing

Stochastic Gradient Descent (SGD) is a preferred optimization method for training large-scale deep learning models. To accelerate the training process, asynchronous SGD (ASGD) has been widely used in a distributed environment, where parameters of the model are maintained by a parameter server called master, and there exist multiple edges known as workers that locally process data in parallel and report the results back to the server. This is also known as edge computing. We propose a new variance reduced ASGD algorithm that reduces the variance of gradient update using the control variate method, and adaptively adjusts the momentum term based on the estimated gradient staleness. We perform experiments on a variety of real datasets to demonstrate the efficiency of the proposed algorithm.