The nonlinear least squares problem has the general form

where r is the function defined by

for some vector-valued function f that maps to .

Least squares problems often arise in data-fitting applications. Suppose that some physical or economic process is modeled by a nonlinear function that depends on a parameter vector x and time t. If is the actual output of the system at time , then the residual

measures the discrepancy between the predicted and observed outputs of the system at time . A reasonable estimate for the parameter x may be obtained by defining the ith component of f by

,

and solving the least squares problem with this definition of f.

From an algorithmic point of view, the feature that distinguishes least squares problems from the general unconstrained optimization problem is the structure of the Hessian matrix of r. The Jacobian matrix of f, can be used to express the gradient of r since Similarly, is part of the Hessian matrix since To calculate the gradient of r, we need to calculate the Jacobian matrix . Having done so, we know the first term in the Hessian matrix without doing any further evaluations. Nonlinear least squares algorithms exploit this structure.

In many practical circumstances, the first term in is more important than the second term, most notably when the residuals are small at the solution. Specifically, we say that a problem has small residuals if, for all x near a solution, the quantities

are small relative to the smallest eigenvalue of .

• Gauss-Newton Method
• Levenberg-Marquardt Method
• Hybrid Methods
• Large Scale Methods
• Techniques for solving L.S. problems with constraints
• Notes and References

Up To:

Unconstrained Optimization.

[ OTC Home Page | NEOS Guide | NEOS Server | Optimization Tree ]

Updated 28 March 1996