Newton's Method: A Nonlinear Optimization Study

Caytuiro Tapia C.; Roberto Pastro C. · Proceedings of the LACCEI international Multi-conference for Engineering, Education and Technology (2024) · DOI: 10.18687/laccei2024.1.1.1160

Resumen

Neural Networks are currently the subject of extensive study and reserch, serving as tools for many works in the academic area. For a neural network to operate properly, it is necessary to carry out a process called training. Training a neural network consists of minimizing the network error. To minimize such error, a first order method can be used. First order methods use a gradient vector of the error function (vector of first derivatives). One of the best-known first-order methods is gradient descent. This method, although functional, can, in many cases, take time to achieve its objective. In such cases it is common to resort to second order methods, such as Newton's Method, which uses a Hessian matrix (matrix of second derivatives). This work presents a comparison of the convergence of the two methods, exploring both aspects of effectiveness and speed of optimization.

Instituciones: Universidad Privada del Norte