A Weighted Linearization Approach to Gradient Descent Optimization

The weighted linearization is a generalization of the first-order Taylor approximation where the computation of the Jacobian matrices at the point of interest is replaced by the computation of the integral of the Jacobian matrix function by a weighting function that expresses how much different parts of the domain should be taken into account during the linearization. This paper blends the weighted linearization with the existing gradient descent (GD) method to develop a novel optimization technique named weighted gradient descent (WGD). The WGD is shown to outperform the GD in terms of mean absolute error, given an appropriate tuning of the WGD hyper-parameters, when applied to various nonlinear functions that are multi-modal in nature, thus exhibiting several optima.