Notes: Iterative Choices of Regularization Parameters in Linear Inverse

Problems
Authors: Kar Kunisch and Jun Zou
Purpose of the Paper:
To investigate the possibility of choosinig more acurate regularization pa-
rameters for OLS in linear inverse problems.
Tools:
Morozov discrepancy principle
Damped Morozov discrepancy principle
Morozovs Principle: The regularization parameter should be chosen so
that the error from regularization is equal to the error in the observation data ie:
||Tf() z

||
2
Y
=
2
Where:
= ||zz

||
Y
- the observation error as measured by the Y norm between
z (the data) and z

, the data with a noise level.

T is a linear operator which maps the parameters which we have measure-
ments for into the solution space.
Damped Morozov Principle: Operates under the same general principle as
the above with the modication:
||Tf() z

||
2
Y
+

||f()||
2
X
=
2
Where:
X denotes another norm
[1, ]
Also note, that if we let , because parameter || < 1 the damped Morozov
principle becomes undamped.
1
Setting Up the Problem The formulation for the inverse problem is:
Tf = z
Where:
T is an operator which takes the measurements we have made, and maps
them into approximations of the solution.
f is our parameter set.
z is the list of solution.
For the purposes of this paper, we are assuming that we are solving the problem
using the OLS method:
Minimize f X :
J(f, ) =
1
2
||Tf z

||
Y
+

2
||f||
2
X
As long as > 0, we have a unique solution to the below system:
Why?
T Tf + f = T z

which has the variational form:

(Tf, Tg)
Y
+ (f, g)
X
= (z

, Tg)
Y
g X
Minimum Value Function: We dene the minimum value function as:
F() = J(f(), ) =
1
2
||Tf() z

||
2
Y
+

2
||f()||
2
X
With derivatives:
F

() =
1
2
||f()||
2
X
F

() = (f(, f

())
X
Lemma 2.3:
The non-negative functoin F() is:
Monotonically increasing
Strictly concave
Heres a picture:
2
Because: , {F

() > 0, F

() < 0}
This result is proved in Lemma 2.3.
Assumption: F(0)
2
/2 F(1). In other words, the value of our noise
squared and divided by two will be somewhere within the range that our func-
tion F() will produce.
Recall that
F() =
1
2
||Tf() z

||
2
Y
+

2
||f()||
2
X
. And:
F

() =
1
2
||f()||
2
X
Then if we take the expression:
F() + (

)F

() =
1
2

1
2
||Tf() z

||
2
Y
+

2
||f()||
2
X
+ (

)
1
2
||f()||
2
X
=
1
2
||Tf() z

||
2
Y
+

2
||f()||
2
X
+ (

)
1
2
||f()||
2
X
IE, we are assuming that there is some kind of linear combination of the mini-
mization function and the derivative such that when we use them to select our
regularization parameter we will get the noise term, squared and divided by 2.
When we make this assumption, then we move to the next assumption:
If F(0) <
1
2

2
F(1), then there exists a unique solution (called

(0, 1] to
our Morozov equation.
Algorithms:
Next, the authors propose that the user employ the Newtons method or a
quasi-Newton method to solve the Morozov equation:
G() = F() + (

)F

()
1
2
= 0
3
In other words, solve the expression between the function and its derivative
so that it is equal to the multiple of the noise term.
Note:
G() = F() + (

)F

()
1
2
= 0
G

() =
1
F

() + (

)F

()
G

() can be expanded to:

1
1
2
||f()||
2
X
+ (

)(f(), f

())
X
Then solving for G

() involves solving for the value of f

() which solves the

equation:
T Tw + w = f()
Newtons Method The general scheme for the Newtons algorithm will be
to:
1. Take a guess between 0 and 1 for
2. Calculate f

(
k
) by solving the expression T Tw + w = f().
3. It should be noted that there is a problem for this: while this will converge
quadratically, it will also require that at each iterationwe solve for f()
and f

(). As a result, we may instead replace the derivative with an

approximation of the derivative:
f
k
(
k
,
k1
) =
f(
k
) f(
k1

k

k1
If we replace the derivative with its approximation, we get a quasi-newton
method.
4
Discussion of the Noise Term: Theorem 3.1:
1. F(0)
1
2

2
F(1)
2.

(0, 1]
Then for a positive constant : I = [

+ ], the sequence of
k
values
produced by the algorithm converge the unique solution

and that it does so

superlinearly.
Model Functions for F(
k
) and F

(
k
)
For the purposes of the paper, the authors represent the above two functions as
model functions. A model function is a paramtrized function which keeps the
major properties of the function F(), and interpolation it. The model function
derived will be an ODE with two parameters that turns out to work fairly well
for the problem:
First:
(Tf(), Tf())
Y
T
1
(f(), f())
X
In other words, we are replacing the inner product of Tf() with itself, with an
approximation using simply a constant T
1
instead of the linear operator T.
This reduces to an ODE:
m

() + m() + T
1
m

() = C
0
m() = C
0
+
C
1
T
1
+
Where the C values are just integration constants. We can specify qualities
for the constants using the following properties for the funtion F():
F() > 0
It is concave IE F

() < 0
These properties are preserved when C
1
< 0 and it is non neative if C
0
+
C1
T1
To reduce this further we have to consider another quality of the above sys-
tem: F(0) = 0. If we use this fact then our model equation m() becomes:
m(0) = F(0) = C
0
+
C
1
T
1
+ 0
= 0
C
1
= T
1
C
0
5
and we can phrase this as the two parameter approximation of
m() = F(
k
) = C
k
{1
T
k
T
k

k
}
and:
m

() = F

(
k
) =
C
k
T
k
(T
k
+
k
)
2
General Algorithm
This algorithm uses the following identies:
F() =
1
2
||Tf() z

||
2
Y
+

2
||f()||
2
X
F

() =
1
2
||f()||
2
X
m(
k
) = F(
k
) = C
k
{1
T
k
T
k

k
}
m

() = F

(
k
) =
C
k
T
k
(T
k
+
k
)
2
T
k
=

2
k
F

(
k
)
F(
k

k
F

(
k
)
C
k
=
F(
k
)
2
F(
k
)
k
F

(
k
)
Steps:
Select a starting guess for
k
where k = 0
1. Calculate F(
k
), F

(
k
)
2. Calculate m()
3. Employ the Newton or quasi Newton method to solve for
k+1
using New-
tons method:
or the quasi-Newton method
4. If |
k+1

k
| tolerance then stop the algorithm. Otherwise, go back to
1.
6
Current Questions:
1. What is f()?
2. Are the two parameter and the newton algorithms dierent?
3. What is the general ordering and structure of the two parameters algo?
To Do:
Investigate the way that the model problems are constructed.
7