Neural Networks: Training with Backpropagation and the Gradient Algorithm

This work involves research and analysis of the topic of learning algorithms of neural networks. This paper presents network learning that is performed by regenerating neural connections. This type of learning uses the back-propagation method. The concept of such a term as the activation function and its features are introduced. Based on some of the given characteristics, a detailed analysis of the choice of each of them for a specific task is provided. The principle of building a neural network using backpropagation and using gradient descent to optimize error detection is considered. The study shows the shortcomings of the backpropagation algorithm and presents methods of backpropagation that eliminate this algorithm's defects. A simple four-layer neural network using backpropagation and gradient descent is investigated to classify handwritten numbers if they were part of the MNIST dataset. The advantages and disadvantages of this type of network are given. The study presents an analysis for better application of the Sigmoid or Relu activation function.