Optimization Toolbox

Levenberg-Marquardt Method

The Levenberg-Marquardt [27],[29] method uses a search direction that is a solution of the linear set of equations

(3-22)

where the scalar controls both the magnitude and direction of . When is zero, the direction is identical to that of the Gauss-Newton method. As tends to infinity, tends toward a vector of zeros and a steepest descent direction. This implies that for some sufficiently large , the term holds true. The term can therefore be controlled to ensure descent even when second order terms, which restrict the efficiency of the Gauss-Newton method, are encountered.

The Levenberg-Marquardt method therefore uses a search direction that is a cross between the Gauss-Newton direction and the steepest descent. This is illustrated in Figure 3-4, Levenberg-Marquardt Method on Rosenbrock's Function. The solution for Rosenbrock's function (Eq. 3-2) converges after 90 function evaluations compared to 48 for the Gauss-Newton method. The poorer efficiency is partly because the Gauss-Newton method is generally more effective when the residual is zero at the solution. However, such information is not always available beforehand, and the increased robustness of the Levenberg-Marquardt method compensates for its occasional poorer efficiency.

Figure 3-4: Levenberg-Marquardt Method on Rosenbrock's Function

Gauss-Newton Method

Nonlinear Least-Squares Implementation