Efficient Convergence Analysis of Stochastic Gradient Descent for Large-Scale Optimization

Stochastic gradient descent (SGD) is an optimization set of rules used to solve huge-scale optimization issues. It works by way of updating an approximate solution, primarily based on the gradient of the goal characteristic with admire to each parameter, in an iterative fashion. SGD is powerful in cases in which the full-gradient updates are not feasible because of the big length of the dataset, on the grounds that the simplest mini-batch of the facts is used for every update. In this paper, a brand new technique for analyzing the convergence homes of SGD is proposed. Particularly, the authors keep in mind the worst-case situation wherein the goal characteristic is composed of parts: a smooth element and a non-clean part, each with its personal Lipschitz constants. The authors then analyze SGD with a hard and fast step size and show that it may converge to the premier answer with a top-quality rate. The proposed approach is efficient in that it requires fewer reviews of the gradient than the prevailing convex optimization strategies. It is predicted that this new theoretical result will facilitate the improvement of extra green and dependable optimization strategies.