Using First-Order Stochastic Based Optimizers in Solving Regression Models

In this research we examine different methods of optimization for Artificial Neural Networks (ANNs) with Deep Learning (DL). We examine the generalization performance of some existing optimizers that exist in the Keras library as Stochastic Gradient Descent (SGD) with and without Nesterov momentum, Adam and AdaMax in solving regression tasks. We use the following activation functions: Rectified Linear Unit (ReLU), Swish, and Sigmoid. We utilize TensorFlow as a backend on a GTX 970 GPU and use a neural network topology based on Deep Learning Multilayered Perceptron (DLMLP) with backpropagation.