Optimization Techniques in Deep Learning Scenarios: An Empirical Comparison

Summary

Machine learning has enormously contributed toward optimization techniques motivating new approaches for optimization algorithms and their usage that have played a pivotal role in data science. The optimization approaches in deep learning has wide applicability with resurgence of novelty starting from Stochastic Gradient Descent to convex and non‐convex and derivative‐free approaches. Selecting an optimizer is an important choice in deep learning scenarios, and the optimization algorithm chosen having convexity principles in their core determines the training speed and final performance predicted by the DL model. The complexity further increases with growing deepness due to hyperparameter tuning and as the datasets become larger and, in turn, they require a fitting optimizer.

In this chapter, we examine the most popular and widely optimizers algorithms in an empirical way. The augmenting behaviors of these are tested on MNIST, SKLEARN datasets. We empirically compare them pointing out their similarities, differences, and likelihood of their suitability for a given applications. Recent variants of optimizers are highlighted with their subtleties. The chapter focuses on the critical role and pinpoints which one would be a better option while making a trade‐off.