A user's guide to nonlinear optimization algorithms

The purpose of this paper is to provide a user's introduction to the basic ideas currently favored in nonlinear optimization routines by numerical analysts. The primary focus will be on the unconstrained problem because the main ideas are much more settled. Although this is not a paper about nonlinear least squares, the rich structure of this important practical problem makes it a convenient example to illustrate the ideas we will discuss. We will make most use of this example in the first three sections which deal with the helpful concept of a local modeling technique and the attendant local convergence analysis. Stress will be put on ways used to improve a poor initial solution estimate, since this is one of the keys to choosing the most suitable routine for a particular application. This material is covered in the rather long Section IV. The discussion of the constrained problem in Section V will be a brief outline of the current issues involved in deciding what algorithms to implement. Section VI is devoted to some concluding remarks including sparse comments on large problems.