L-Curve Method

One of the most common approaches to estimating the optimal regularization parameter is the L-curve method. This method utilizes two values: the
regularized solution ||f ()||2Y to the least squares problem:
min J(f ()) + ||f ()||2X

f X

and the residual : | |T f () z ||2Y .

An observation due to [1](C. L. Lawson and R. J. Hanson, Solving Least
Squares Problems), is that when the regularization parameter is too small,
the value for ||f ()||X will decrease slowly, while | |T f () z ||2Y will increase
more quickly by comparison. As becomes larger, the increase in the residual
becomes proportionally more rapid. When the logarithms of both quantities are
plotted for a range of regularization parameters, a distinct L shape appears
which lends its name to the method.
Figure 1: L-Curve for Shaw Problem

The L-curve method seeks to find the most appropriate regularization parameter by finding the point at which the regularization parameter begins to
have a larger effect on the change in the residual. Heuristically, this can be
done by simply performing a plot like the one above and estimating the point
at which the greatest curvature is attained. Estimating the point of greatest
curvature can be inadequate when a very precise measurement is required, or
when the loglog plot yields no obvious curve.

The point of greatest curvature can be found more precisely by defining the
curvature as a function of the log values of the residual, and the solution norms:
R() = log | |T f () z ||2Y
S() = log||f ()||2Y
R(), S() can be recovered by substituting the singular value decomposition of operator T into | |T f () z ||2Y and ||f ()||2Y making the computation
of f () and | |T f () z ||2Y a straight forward task. From the singular value
decomposition due to Hansen [3] we can express ||f ()||2Y and | |T f () z ||2Y as
being made up of unitary matrices u, v , and singular values :
||f ()||2Y =

N
X
i=1

| |T f ()

z ||2Y =

Fi

uTi z
i

N
X

(1 Fi )uTi z)2

i=1

For data matrix z and filter factors F calculated as:

Fi =

i2
i2 + 2

(1)

Differentiation in terms of produces:

s0 () =

N
4X
(uT z)2
(1 Fi )Fi2 i 2
i=1
i

r0 () =

N
4X
(1 Fi )Fi (uTi z)2
i=1

We calculate the second derivative of the solution and residual functionals

to get:
s00 () = ...
r00 () = 2S 0 () 2 S 00 ()
Since we are considering the logarithms of the functions, we differentiate the
logarithmic functions to obtain:
s0 ()
s()
00
0
2
s
()s()

(s
())
S 00 () =
s()
r0 ()
R0 () =
r()
00
0
2
r
()r()

(r
())
R00 () =
r()
S 0 () =

Through a derivation originally discussed in , the derivatives of these functions in respect to can then be taken, and used to find the curvature for a
particular :
R00 ()S 0 () R0 ()S 00 ()
C() =
((R0 ()2 + S 0 ()2 )1.5
To find the point of maximum curvature C(L ), the negative curvature is
calculated over a range of regularization parameters. The estimate producing
the lowest value is initialized for a bounded minimization algorithm to identify
L .

Second Order Example: The second order example was implemented

with a grid size of 20 with measurements of the curvature being taken at onethousand points on [4.8032 105 , .025518]. We tested the L-curve algorithm
over a range weighted sinusoidal noise from = 0 to .1 :
Figure 2: L-Curve and Curvature for = 0

Figure 3: L-Curve and Curvature for = .002

Figure 4: L-Curve and Curvature for = .004

Figure 5: L-Curve and Curvature for = .006

Figure 6: L-Curve and Curvature for = .008

Figure 7: L-Curve and Curvature for = .01

Figure 8: L-Curve and Curvature for = .03

Figure 9: L-Curve and Curvature for = .05

10

Figure 10: L-Curve and Curvature for = .07

11

Figure 11: L-Curve and Curvature for = .1

12

Our distinctive L-curve mentioned at the beginning of the section were noticeably absent. Furthermore, the L-curve method was noted to have produced
the A heuristic approach would be inadequate, however the use of an optimization algorithm allowed the identification of C(L ):

0
0.002
0.004
0.006
0.008
0.01
0.03
0.05
0.07
0.1

L
4.8148e-05
0.00012778
0.00017826
0.00021254
0.00024404
0.00027325
0.00085718
0.0021447
0.0027401
0.003457

L2 Error
15.3988
24.733
31.0852
35.6234
39.8981
43.9161
114.7885
194.086
213.3111
229.8098

S(L )
79.1448
78.6185
78.5561
78.7618
79.1294
79.6198
82.4843
80.1575
80.9815
82.1937

R(L )
0.00011467
0.00065215
0.0014005
0.0022955
0.0033506
0.0045334
0.028831
0.079813
0.12566
0.19809

Fourth Order Example

The L-curve method was applied to the fourth order problem in the same
manner as the second order problem, with sinusoidal noise weighted from 0 to
.1.
The curvature for the fourth order problems were much more noticeable.

13

Figure 12: L-Curve and Curvature for = 0

14

Figure 13: L-Curve and Curvature for = .002

15

Figure 14: L-Curve and Curvature for = .004

16

Figure 15: L-Curve and Curvature for = .006

17

Figure 16: L-Curve and Curvature for = .008

18

Figure 17: L-Curve and Curvature for = .01

19

Figure 18: L-Curve and Curvature for = .03

20

Figure 19: L-Curve and Curvature for = .05

21

Figure 20: L-Curve and Curvature for = .07

22

Figure 21: L-Curve and Curvature for = .1

0
0.002
0.004
0.006
0.008
0.01
0.03
0.05
0.07
0.1

L
2.7229e-08
5.8595e-06
1.1519e-05
1.644e-05
2.0351e-05
2.3884e-05
4.531e-05
5.413e-05
6.0227e-05
6.701e-05

L2 Error
1655003410.2311
1711616019.9866
1711844287.1125
1711917747.7788
1711951231.2087
1711972193.9675
1712029789.9674
1712040240.9815
1712045679.7016
1712050601.655

S(L )
205929.5982
185693.4554
181325.5229
179209.5255
177976.8264
177022.131
172891.1245
173285.0142
174715.1568
177865.0406

R(L )
0.0032037
1.3386
2.677
4.0127
5.3457
6.6791
20.0005
33.2813
46.5598
66.477

References

[1] C. L. Lawson and R. J. Hanson, Solving Least Squares Problems

23