On Faster Convergence of Scaled Sign Gradient Descent

Communication has been seen as a significant bottleneck in industrial applications over large-scale networks. To alleviate the communication burden, sign-based optimization algorithms have gained popularity recently in both industrial and academic communities, which is shown to be closely related to adaptive gradient methods, such as Adam. Along this line, this article investigates faster convergence for a variant of sign-based gradient descent, called scaled
sign
GD, in three cases: First, the objective function is strongly convex; second, the objective function is nonconvex but satisfies the Polyak–Łojasiewicz inequality; third the gradient is stochastic, called scaled
sign
SGD in this case. For the first two cases, it can be shown that the scaled
sign
GD converges at a linear rate. For case third, the algorithm is shown to converge linearly to a neighborhood of the optimal value when a constant learning rate is employed, and the algorithm converges at a rate of
$O(1/k+1/k^{2}+1/k^{3})$
when using a diminishing learning rate, where
$k$
is the iteration number. The results are also extended to the distributed setting by majority vote in a parameter-server framework. Finally, numerical experiments are performed to corroborate the theoretical findings.