Evolution and Recent Trends for the SGD Algorithm: Key Takeaways from an Educational Short Course

One of the key challenges when designing a ten (10) hours educational short course, entitled “A Hands-on Approach for Implementing Stochastic Optimization Algorithms from Scratch”, which was accepted for inclusion at ICASSP'23, was related to addressing “how to introduce the Stochastic Gradient Descent (SGD) algorithm and variants in a consistent, accessible fashion”? From a simplistic perspective, the SGD algorithm is nothing else than the classical gradient descent (GD) algorithm along with a (very) noisy gradient. Nonetheless, arguably, SGD's most influential variants, e.g. AdaGrad, RMSprop and Adam, nor more recent ones (LookAhead, E-Adam, MadGrad, among several others) may not be explained in such superficial terms. Moreover, such variants are usually given as as black-boxes by most deep-learning (DL) libraries (e.g. TensorFlow, PyTorch, etc.). In this article, based on the experience of the aforementioned short-course, I propose to link the SGD algorithm and variants via an “evolutionary path”, in which each SGD variant may be understood as a set of add-on features over the vanilla SGD, resulting in a generalized algorithm along with a “family tree” graph which are both intuitive and useful when implementing a given SGD variant.