Node-Decoupled Extended Kalman Filter versus Adam Optimizer in Approximation of Functions with Multilayer Neural Networks

The training algorithm is a determining factor in the success of artificial neural networks. Until now, first-order optimizers based on gradient descent are the most used in neural network applications. However, strategies that involve second-order information have been investigated with important results. In this article, a training algorithm based on the Node Decoupled Extended Kalman Filter (NDEKF) is reviewed and compared with one of the most widely used algorithms, Adam optimizer. Two experiments with multilayer neural networks are carried out in the task of approximation of nonlinear functions. In the performance tests of the algorithms, both the speed of convergence and the precision of the approximation are compared, for the same number of epochs in the training process. Results are presented, in which the NDEKF stands out.