Chapter 6

Linear operators and adjoints

Contents
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1
Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2
Spaces of bounded linear operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3
Inverse operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5
Linearity of inverses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5
Banach inverse theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5
Equivalence of spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5
Isomorphic spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6
Isometric spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6
Unitary equivalence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7
Adjoints in Hilbert spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.11
Unitary operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.13
Relations between the four spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.14
Duality relations for convex cones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.15
Geometric interpretation of adjoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.15
Optimization in Hilbert spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.16
The normal equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.16
The dual problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.17
Pseudo-inverse operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.18
Analysis of the DTFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.21
6.1
Introduction
The eld of optimization uses linear operators and their adjoints extensively.
Example. differentiation, convolution, Fourier transform, Radon transform, among others.
Example. If A is a n m matrix, an example of a linear operator, then we know that y Ax
2
is minimized when x =
[A

A]
1
A

y.
We want to solve such problems for linear operators between more general spaces. To do so, we need to generalize transpose
and inverse.
6.1
6.2 c J. Fessler, December 21, 2004, 13:3 (student version)
6.2
Fundamentals
We write T : D Y when T is a transformation from a set D in a vector space X to a vector space Y.
The set D is called the domain of T. The range of T is denoted
R(T) = {y Y : y = T(x) for x D} .
If S D, then the image of S is given by
T(S) = {y Y : y = T(s) for s S} .
If P Y, then the inverse image of P is given by
T
1
(P) = {x D : T(x) P} .
Notation: for a linear operator A, we often write Ax instead of A(x).
For linear operators, we can always just use D = X, so we largely ignore D hereafter.
Denition. The nullspace of a linear operator A is N(A) = {x X : Ax = 0} .
It is also called the kernel of A, and denoted ker(A).
Exercise. For a linear operator A, the nullspace N(A) is a subspace of X.
Furthermore, if A is continuous (in a normed space X), then N(A) is closed [3, p. 241].
Exercise. The range of a linear operator is a subspace of Y.
Proposition. A linear operator on a normed space X (to a normed space Y) is continuous at every point X if it is continuous
at a single point in X.
Proof. Exercise. [3, p. 240].
Luenberger does not mention that Y needs to be a normed space too.
Denition. A transformation T from a normed space X to a normed space Y is called bounded iff there is a constant M such that
T(x) Mx , x X.
Denition. The smallest such M is called the norm of T and is denoted ||| T |||. Formally:
||| T ||| inf {M R : T(x) Mx , x X} .
Consequently:
T(x)
Y
||| T ||| x
X
, x X.
Fact. For a linear operator A, an equivalent expression (used widely!) for the operator norm is
|||A||| = sup
x1
Ax .
Fact. If X is the trivial vector space consisting only of the vector 0, then |||A||| = 0 for any linear operator A.
Fact. If X is a nontrivial vector space, then for a linear operator A we have the following equivalent expressions:
|||A||| = sup
x=0
Ax
x
= sup
x=1
Ax .
c J. Fessler, December 21, 2004, 13:3 (student version) 6.3
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Example. Consider X = (R
n
,

) and Y = R.
Clearly Ax = a
1
x
1
+ +a
n
x
n
so Ax
Y
= |Ax| = |a
1
x
1
+ +a
n
x
n
| |a
1
||x
1
|+ +|a
n
||x
n
| (|a
1
|+ +|a
n
|) x

.
In fact, if we choose x such that x
i
= sgn(a
i
), then x

= 1 we get equality above. So we conclude |||A||| = |a

1
| + +|a
n
|.
Example. What if X =
_
R
n
,
p
_
? ??
Proposition. A linear operator is bounded iff it is continuous.
Proof. Exercise. [3, p. 240].
More facts related to linear operators.
If A : X Y is linear and X is a nite-dimensional normed space, then A is continuous [3, p. 268].
If A : X Y is a transformation where X and Y are normed spaces, then A is linear and continuous iff A(

i=1

i
x
i
) =

i=1

i
A(x
i
) for all convergent series

i=1

i
x
i
. [3, p. 237].
This is the superposition principle as described in introductory signals and systems courses.
Spaces of bounded linear operators
Denition. If T
1
and T
2
are both transformations with a common domain X and a common range Y, over a common scalar eld,
then we dene natural addition and scalar multiplication operations as follows:
(T
1
+T
2
)(x) = T
1
(x) +T
2
(x)
(T
1
)(x) = (T
1
(x)).
Lemma. With the preceding denitions, when X and Y are normed spaces the following space of operators (!) is a vector space:
B(X, Y) = {bounded linear transformations from X to Y} .
(The proof that this is a vector space is within the next proposition.)
This space is analogous to certain types of dual spaces (see Ch. 5).
Not only is B(X, Y) a vector space, it is a normed space when one uses the operator norm |||A||| dened above.
Proposition. (B(X, Y), ||| |||) is a normed space when X and Y are normed spaces.
Proof. (sketch)
Claim 1. B(X, Y) is a vector space.
Suppose T
1
, T
2
B(X, Y) and F.
(T
1
+T
2
)(x)
Y
= T
1
(x) +T
2
(x)
Y
|| T
1
(x)
Y
+ T
2
(x)
Y
||||| T
1
||| x
X
+ ||| T
2
||| x
X
= Kx
X
, where
K ||||| T
1
||| +||| T
2
|||. So T
1
+T
2
is a bounded operator. Clearly T
1
+T
2
is a linear operator.
Claim 2. ||| ||| is a norm on B.
The hardest part is verifying the triangle inequality:
||| T
1
+T
2
||| = sup
x=1
(T
1
+T
2
)x
Y
sup
x=1
T
1
x
Y
+ sup
x=1
T
2
x
Y
= ||| T
1
||| +||| T
2
|||.
2
Are there other valid norms for B(X, Y)? ??
Remark. We did not really need linearity in this proposition. We could have shown that the space of bounded transformations
from X to Y with ||| ||| is a normed space.
6.4 c J. Fessler, December 21, 2004, 13:3 (student version)
Not only is B(X, Y) a normed space, but it is even complete if Y is.
Theorem. If X and Y are normed spaces with Y complete, then (B(X, Y), ||| |||) is complete.
Proof.
Suppose {T
n
} is a Cauchy sequence (in B(X, Y)) of bounded linear operators, i.e., |||T
n
T
m
||| 0 as n, m .
Claim 0. x X, the sequence {T
n
(x)} is Cauchy in Y.
T
n
(x) T
m
(x)
Y
= (T
n
T
m
)(x)
Y
|||T
n
T
m
||| x
X
0 as n, m .
Since Y is complete, for any x X the sequence {T
n
(x)} converges to some point in Y. (This is called pointwise convergence.)
So we can dene an operator T : X Y by T(x) lim
n
T
n
(x).
To show B is complete, we must rst show T B, i.e., 1) T is linear, 2) T is bounded.
Then we show 3) T
n
T (convergence w.r.t. the norm ||| |||).
Claim 1. T is linear
T(x +z) = lim
n
T
n
(x +z) = lim
n
[T
n
(x) +T
n
(z)] = T(x) +T(z) .
(Recall that in a normed space, if u
n
u and v
n
v, then u
n
+v
n
u +v.)
Claim 2: T is bounded
Since {T
n
} is Cauchy, it is bounded, so K < s.t. |||T
n
||| K, n N. Thus, by the continuity of norms, for any x X:
T(x)
Y
= lim
n
T
n
(x)
Y
lim
n
|||T
n
||| x
X
Kx
X
.
Claim 3: T
n
T
Since {T
n
} is Cauchy,
> 0, N N s.t. n, m > N = |||T
n
T
m
|||
= T
n
(x) T
m
(x)
Y
x
X
, x X
= lim
m
T
n
(x) T
m
(x)
Y
x
X
, x X
= T
n
(x) T(x)
Y
x
X
, x X by continuity of the norm
= |||T
n
T ||| .
We have shown that every Cauchy sequence in B(X, Y) converges to some limit in B(X, Y), so B(X, Y) is complete. 2
Corollary. (B(X, R), ||| |||) is a Banach space for any normed space X.
Why? ??
We write A B(X, Y) as shorthand for A is a bounded linear operator from normed space X to normed space Y.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Denition. In general, if S : X Y and T : Y Z, then we can dene the product operator or composition as a
transformation T S : X Z by (T S)(x) = T(S(x)).
Proposition. If S B(X, Y) and T B(Y, Z), then T S B(X, Z).
Proof. Linearity of the composition of linear operators is trivial to show.
To show that the composition is bounded: T Sx
Z
||| T ||| Sx
Y
||| T ||||||S||| x
X
. 2
Does it follow that ||| T S||| = ||| T ||||||S|||? ??
c J. Fessler, December 21, 2004, 13:3 (student version) 6.5
6.3
Inverse operators
Denition. T : X Y is called one-to-one mapping of X into Y iff x
1
, x
2
X and x
1
= x
2
=T(x
1
) = T(x
2
).
Equivalently, T is one-to-one if the inverse image of any point y Y is at most a single point in X, i.e.,

T
1
({y})

1, y Y.
Denition. T : X Y is called onto iff T(X) = Y. This is a stronger condition than into.
Fact. If T : X Y is one-to-one and onto Y, then T has an inverse denoted T
1
such that T(x) = y iff T
1
(y) = x.
Many optimization methods, e.g., Newtons method, require inversion of the Hessian matrix (or operator) corresponding to a cost
function.
Lemma. [3, p. 171]
If A : X Y is a linear operator between two vector spaces X and Y, then A is one-to-one iff N(A) = {0}.
Linearity of inverses
We rst look at the algebraic aspects of inverse operators in vector spaces.
Proposition. If a linear operator A : X Y (for vector spaces X and Y) has an inverse, then that inverse A
1
is also linear.
Proof. Suppose A
1
(y
1
) = x
1
, A
1
(y
2
) = x
2
, A(x
1
) = y
1
, and A(x
2
) = y
2
. Then by the linearity of Awe have A(x
1
+x
2
) =
Ax
1
+Ax
2
= y
1
+y
2
, so A
1
(y
1
+y
2
) = x
1
+x
2
= A
1
(y
1
) +A
1
(y
2
). 2
6.4
Banach inverse theorem
Now we turn to the topological aspects, in normed spaces.
Lemma. (Baire) A Banach space X is not the union of countably many nowhere dense sets in X.
Proof. see text
Theorem. (Banach inverse theorem)
If A is a continuous linear operator from a Banach space X onto a Banach space Y for which the inverse operator A
1
exists,
then A
1
is continuous.
Proof. see text
Combining with the earlier Proposition that linear operators are bounded iff they are continuous yields the following.
Corollary.
X, Y Banach and A B(X, Y) and A invertible =A
1
B(Y, X)
Equivalence of spaces (one way to use operators)
in vector spaces
in normed spaces
in inner product spaces
spaces relation operator requirements
vector isomorphic isomorphism linear, onto, 1-1
normed topologically isomorphic topological isomorphism linear, onto, invertible, continuous
normed isometrically isomorphic isometric isomorphism linear, onto, norm preserving =1-1, continuous
inner product unitarily equivalent unitary isometric isomorphism that preserves inner products
6.6 c J. Fessler, December 21, 2004, 13:3 (student version)
Isomorphic spaces
Denition. Vector spaces X and Y are called isomorphic (think: same structure) iff there exists a one-to-one, linear mapping
T of X onto Y.
In such cases the mapping T is called an isomorphism of X onto Y.
Since an isomorphism T is one-to-one and onto, T is invertible, and by the linear of inverses proposition in 6.3, T
1
is linear.
Example. Consider X = R
2
and Y = {f(t) = a +bt on [0, 1] : a, b R}.
An isomorphism is f = T(x) f(t) = a +bt where x = (a, b), with inverse T
1
(f) = (f(0), f(1) f(0)).
Exercise. Any real n-dimensional vector space is isomorphic to R
n
(problem 2.14 p. 44) [3, p. 268].
However, they need not be topologically isomorphic [3, p. 270].
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
So far we have said nothing about norms. In normed spaces we can have a topological relationship too.
Denition. Normed spaces X and Y are called topologically isomorphic iff there exists a continuous linear transformation T of
X onto Y having a continuous inverse T
1
. The mapping T is called a topological isomorphism.
Theorem. [3, p. 258] Two normed spaces are topologically isomorphic iff there exists a linear transformation T with domain
X and range Y and positive constants m and M s.t. mx
X
T x
Y
Mx
X
, x X.
Example. In the previous example, consider (X,

) and (Y,
2
). Then for the same T described in that example:
x = (a, b) =T(x)
2
Y
=
_
1
0
(a +bt)
2
dt = a
2
+ab +b
2
/3 a
2
+|a| |b| +b
2
/3 (1 + 1 + 1/3) x
2

= 7/3 x
2

.
Also T(x)
2
Y
= a
2
+ab +b
2
/3 = (a +b/2)
2
+b
2
/12 = (a

3/2 +b/

3)
2
+a
2
/4 x
2

/12.
So X and Y are topologically isomorphic for the given norms.
Exercise. (C[0, 1],

) and (C[0, 1],

1
) are not topologically isomorphic. Why? ??
Isometric spaces
Denition. Let X and Y be normed spaces. A mapping T : X Y is called norm preserving iff T(x)
Y
= x
X
, x X.
In particular, if T is norm preserving, then ||| T ||| = 1. What about the converse? ??
Proposition. If T is linear and norm preserving, then T is one-to-one, i.e., T(x) = T(z) =x = z.
Proof. If T(x) = y and T(z) = y, then by linearity T(x z) = 0.
So since T is norm preserving, x z = 0 = 0, so x = z. 2
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Denition. If T : X Y is both linear and norm preserving, then T is called a linear isometry.
If, in addition, T is onto Y, then X and Y are called isometrically isomorphic, and T is called an isometric isomorphism.
Remark. To illustrate why we require onto here, consider T : E
n

2
dened by T(x) = (x
1
, . . . , x
n
, 0, 0, . . .).
This T is linear, one-to-one, and norm preserving, but not onto.
Exercise. Every normed space X is isometrically isomorphic to a dense subset of a Banach space

X.
(problem 2.15 on p. 44)
Normed spaces that are isometrically isomorphic can, in some sense, be treated as being identical, i.e., they have identical proper-
ties. However, the specic isomorphism can be important sometimes.
Example. Consider Y =
p
(N) = {(a
1
, a
2
, . . .) :

i=1
|a
i
|
p
< }
and X =
p
(Z) =
_
(. . . , a
2
, a
1
, a
0
, a
1
, a
2
, . . .) :

i=
|a
i
|
p
<
_
. Dene the mapping T : X Y by y = (b
1
, b
2
, . . .) =
T(x) if b
i
= a
z(i)
where z(i) = (1)
i
i/2 . Note that z : {1, 2, 3, . . .} 0, 1, 1, 2, 2, . . ..
This mapping T is clearly an isometric isomorphism, so
p
(Z) and
p
(N) are isometrically isomorphic. Hence we only bother
to work with
p
=
p
(N) since we know all algebraic and topological results will carry over to double-sided sequences.
c J. Fessler, December 21, 2004, 13:3 (student version) 6.7
Unitary equivalence in inner product spaces
Denition. Two inner product spaces X and Y are unitarily equivalent iff there is an isomorphism U : X Y of X onto Y
that preserves inner products: Ux
1
, Ux
2

Y
= x
1
, x
2

X
, x
1
, x
2
X. The mapping U is called a unitary operator.
Fact. If U is unitary, then U is norm preserving since Ux = Ux, Ux = x, x = x . Clearly |||U||| = 1.
Furthermore, since U is onto, U is an isometric isomorphism. Interestingly, the converse is also true [3, p. 332].
Theorem. A mapping U of X onto Y, where X and Y are inner product spaces, is an isometric isomorphism iff U is a
unitary operator.
Proof. (=) was just argued above.
(=) Suppose U is an isometric isomorphism.
Using the parallelogram law, the linearity of U, and the fact that Ux = x, we have:
4 Ux, Uy = Ux +Uy
2
Ux Uy
2
+i Ux +iUy
2
i Ux iUy
2
= U(x +y)
2
U(x y)
2
+i U(x +iy)
2
i U(x iy)
2
= x +y
2
x y
2
+i x +iy
2
i x iy
2
= 4 x, y .
Thus U an isometric isomorphism =U unitary. 2
Remark. After dening adjoints we will show that U
1
= U

in Hilbert spaces.
Exercise. Any complex n-dimensional inner product space is unitarily equivalent to C
n
[3, p. 332].
Every separable Hilbert space is unitarily equivalent with
2
or some C
n
[3, p. 339].
Example. Continue the previous f(t) = a +bt example, but now use E
n
and (Y, ,
2
). If g(t) = c +dt then
T x
1
, T x
2
= f, g =
_
1
0
f(t)g(t) dt =
_
1
0
(a+bt)(c+dt) dt = ac+
ad
2
+
bc
2
+
bd
3
= [a b]
_
1 1/2
1/2 1/3
_ _
c
d
_
= x

1
Gx
2
,
so we dene U = T G
1/2
, then
Ux
1
, Ux
2
= T G
1/2
x
1
, T G
1/2
x
2
= (G
1/2
x
1
)

G(G
1/2
x
1
) = x

1
x
1
= x
1
, x
2
.
Example. The Fourier transform F() =
_

f(t) e
2t
dt is a unitary mapping of L
2
[R] onto itself.
Example. Soon we will analyze the discrete-time Fourier transform (DTFT) operator, dened by
G = Fg G() =

n=
g
n
e
n
.
We will show that F B(
2
, L
2
[, ]) and F is invertible. And Parsevals relation from Fourier analysis is that
_
1

2
Fg,
1

2
Fh
_
=
1
2
_

G()H

() d =

n=
g
n
h

n
= g, h .
So
2
and L
2
[, ] are unitarily equivalent, and the unitary operator needed is simply U =
1

2
F.
6.8 c J. Fessler, December 21, 2004, 13:3 (student version)
The following extension theorem is useful in proving that every separable Hilbert space is isometrically isomorphic to
2
.
Theorem. Let X be a normed space, Y a Banach space, and M X and N Y be subspaces.
Suppose that
M = X, and N = Y,

T : M N is a bounded linear operator.
Then there exists a unique linear operator T : X Y such that T |M =

T. Moreover, ||| T ||| = |||

T |||.
If, in addition,
X is a Banach space,

T
1
: N M exists and is bounded,
then T is also onto Y.
Proof. Note: T |M reads T restricted to M.
T |M =

T means T(x) =

T(x) for all x M.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Claim 1. If {x
n
} M is Cauchy in M, then
_

T(x
n
)
_
is Cauchy in N.
_
_
_

T(x
n
)

T(x
m
)
_
_
_
Y
=
_
_
_

T(x
n
x
m
)
_
_
_
Y
|||

T ||| x
n
x
m

X
0 as n, m .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Claim 2. For {x
n
} , { x
n
} M, suppose x
n
x X and x
n
x.
Then
_

T(x
n
)
_
and
_

T( x
n
)
_
both converge and to the same limit.
_

T(x
n
)
_
and
_

T( x
n
)
_
are both Cauchy in N by Claim 1 since x
n
and x
n
both converge.
Since Y is complete, y, y Y s.t.

T(x
n
) y and

T( x
n
) y.
But y y
Y
= lim
n
_
_
_

T(x
n
)

T( x
n
)
_
_
_
Y
= lim
n
_
_
_

T(x
n
x
n
)
_
_
_
Y
lim
n
|||

T ||| x
n
x
n

X
= |||

T ||| x x
X
= 0, using norm continuity, linearity of

T, and boundedness of

T.
Now we dene T : X Y as follows.
For x X = M, {x
n
} M such that x
n
x, so we dene T(x) lim
n

T(x
n
).
By Claim 2, T is well-dened, and moreover, if x M, then T(x) =

T(x), i.e., T |M =

T.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Claim 3. ||| T ||| = |||

T |||
x X, T(x) = lim
n

T(x
n
) where x
n
M and x
n
x.
Thus T(x)
Y
= lim
n
_
_
_

T(x
n
)
_
_
_
Y
lim
n
|||

T ||| x
n

X
= |||

T ||| x
X
. Thus ||| T ||| |||

T |||.
However, x M, T(x) =

T(x) =||| T ||| |||

T |||, since T is dened on X M.
Thus ||| T ||| = |||

T |||.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Claim 4. T is linear, which is trivial to prove.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Claim 5. (uniqueness of T) If L
1
, L
2
B(X, Y), then L
1
|M = L
2
|M =L
1
= L
2
.
For x X = M, {x
n
} M s.t. x
n
x. Thus, by the continuity of L
1
and L
2
,
L
1
(x) = lim
n
L
1
(x
n
) = lim
n
L
2
(x
n
) = L
2
(x), since L
1
(x
n
) = L
2
(x
n
) by L
1
|M = L
2
|M.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Claim 6. If

T
1
: N M exists and is bounded, and if X is a Banach space, then T is onto Y.
Let y Y. Since N = Y, {y
n
} N s.t. y
n
y.
By the same reasoning as in Claim 1, x
n
=

T
1
(y
n
) is Cauchy in X.
Since X is a Banach space, x X s.t. x
n
x.
Thus T(x) = lim
n

T(x
n
) = lim
n

T
_

T
1
(y
n
_
) = lim
n
y
n
= y.
Thus, y Y, x X s.t. T(x) = y. 2
c J. Fessler, December 21, 2004, 13:3 (student version) 6.9
The following theorem is an important application of the previous theorem.
Theorem. Every separable Hilbert space H is isometrically isomorphic to
2
.
Proof.
H separable = a countable orthonormal basis {e
i
}. (Homework.) Let M = [{e
i
}

i=1
].
Of course
2
has a countable orthonormal basis { e
i
}, where e
ij
=
ij
(Kronecker). Let N = [{ e
i
}

i=1
].
Dene

T : M N to be the linear operator for which

T(e
i
) = e
i
.
(Exercise. Think about why the hereconsider M.)
Then since any x M has form x =

n
i=1
c
i
e
i
for some n N:
_
_
_

T(x)
_
_
_
2
=
_
_
_
_
_

T
_
n

i=1
c
i
e
i
__
_
_
_
_
2
=
_
_
_
_
_
n

i=1
c
i

T(e
i
)
_
_
_
_
_
2
=
_
_
_
_
_
n

i=1
c
i
e
i
_
_
_
_
_
2
=

_
n

i=1
|c
i
|
2
=
_
_
_
_
_
n

i=1
c
i
e
i
_
_
_
_
_
H
= x ,
(6-1)
so |||

T ||| = 1 on M.
Clearly

T
1
exists on N and is given by

T
1
( e
i
) = e
i
, so |||

T ||| = 1 on N by a similar argument.
Since
2
is a Banach space, we have established all the conditions of the preceding theorem, so there exists a
unique linear operator T from H to
2
that is onto
2
with ||| T ||| = |||

T ||| = 1.
Since T is bounded, it is continuous, so one can take limits as n in (6-1) to show that T x = x ,
so T is norm preserving.
Thus T is an isometric isomorphism for H and
2
, so H and
2
are isometrically isomorphic. 2
Corollary. L
2
is isometrically isomorphic to
2
.
A remarkable result. (Perhaps most of what is remarkable here is that L
2
is separable.)
A practical consequence: usually one can search for counterexamples in
2
(and its relatives) rather than L
2
.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Example. The (different) spaces of odd and even functions in L
2
[, ] are isometrically isomorphic.
Example. Elaborating. Let H = L
2
[, ] and dene
e
2k
= cos(kt) /c
2k
e
2k+1
= sin(kt) /c
2k+1
, k = 0, 1, 2, . . .
c
2k
=
_

cos
2
(kt) dt c
2k+1
=
_

sin
2
(kt) dt
X = [{e
2k
}] = {f L
2
: f even} Y = [{e
2k+1
}] = {f L
2
: f odd} .
Then H, X, and Y are each Banach spaces that are isometrically isomorphic to each other!
And each is isometrically isomorphic to
2
.
Example.
2
is isometrically isomorphic to L
2
[, ]. (Just use the DTFT.)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
But an even stronger result holds than the above...
Every separable Hilbert space is unitarily equivalent with
2
or some C
n
[3, p. 339].
6.10 c J. Fessler, December 21, 2004, 13:3 (student version)
Summary
Insert 5.1-5.3 here!
c J. Fessler, December 21, 2004, 13:3 (student version) 6.11
6.5
Adjoints in Hilbert spaces
When performing optimization in inner product spaces, often we need the transpose of a particular linear operator. The term
transpose only applies to matrices. The more general concept for linear operators is called the adjoint.
Luenberger presents adjoints in terms of general normed spaces. In my experience, adjoints most frequently arise in inner product
spaces, so in the interest of time and simplicity these notes focus on that case. The treatment here generalizes somewhat the
treatment in Naylor [3, p. 352].
Recall the following fact from linear algebra. If A : C
n
C
m
is a mn matrix, then
Ax, y
C
n = y

Ax = (A

y)

x = x, A

y
C
m .
This section generalizes the above relationship to general Hilbert spaces.
Let A B(X, Y) where X and Y are Hilbert spaces.
Let y Y be a xed vector, and consider the following functional:
g
y
: X C, where g
y
(x) Ax, y
Y
.
g
y
is clearly linear, since A is linear and , y
Y
is linear.
|g
y
(x)| Ax
Y
y
Y
|||A||| x
X
y
Y
by Cauchy-Schwarz and since A is bounded.
Thus g
y
is bounded, with |||g
y
||| |||A||| y
Y
.
In other words, g
y
X

.
Denition. By the Riesz representation theorem (here is where we use completeness), for each such y Y there exists a unique
z = z
y
X such that
g
y
(x) = x, z
y

X
, x X.
So we can dene legitimately a mapping A

: Y X, called the adjoint of A, by the relationship z

y
= A

(y).
The dening property of A

is then:
Ax, y
Y
= x, A

(y)
X
, x X, y Y.
(At this point we should write A

(y) rather than A

y since we have not yet shown A

is linear, though we will soon.)

Lemma. A

is the only mapping of Y to X that satises the preceding equality.

Proof. For any y Y, suppose x X we have Ax, y
Y
= x, T
1
(y)
X
= x, T
2
(y)
X
.
Then 0 = x, T
1
(y) T
2
(y)
X
so T
1
(y) = T
2
(y), y Y. 2
Exercise. Here are some simple facts about adjoints, all of which concur with those of Hermitian transpose in Euclidean space.
x, y = x,

y so for A : X X dened by Ax = x we have A

y =

y. So reuse of the asterix is acceptable.

I

= I , 0

XY
= 0
YX
A

= A (see Thm below)

(ST)

= T

(S +T)

= S

+T

(A)

Note: these last two properties are unrelated to the question of whether A

is a linear operator! ??
Example. Consider X = L
2
[0, 1], Y = C
2
, and A : X Y is dened by
y = Ax y
1
= [Ax]
1
=
_
1
0
tx(t) dt, y
2
= [Ax]
2
=
_
1
0
t
2
x(t) dt .
We can guess that the adjoint of A is dened by x = A

y x(t) = (A

y)(t) = y
1
t +y
2
t
2
.
This is veried easily:
Ax, y
Y
= y

1
[Ax]
1
+y

2
[Ax]
2
= y

1
_
1
0
tx(t) dt +y

2
_
1
0
t
2
x(t) dt =
_
1
0
x(t)
_
y
1
t +y
2
t
2
. .
(A

y)(t)
_

dt = x, A

y
X
.
6.12 c J. Fessler, December 21, 2004, 13:3 (student version)
Theorem. Suppose A B(X, Y) where X and Y are Hilbert spaces.
The adjoint operator A

is linear and bounded, i.e., A

B(Y, X).
|||A

||| = |||A|||
(A

= A
Proof.
Claim 1. A

is linear.
x, A

(y +z)
X
= Ax, y +z
Y
=

Ax, y
Y
+Ax, z
Y
=

x, A

(y)
X
+x, A

(z)
X
= x, A

(y) +A

(z)
X
, which holds for all x X and y Y, so A

(y +z) = A

(y) +A

(z) by the usual Lemma once

again.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Claim 2. A

is bounded and |||A

||| |||A|||.
A

y
2
X
= A

y, A

y
X
= AA

y, y
Y
AA

y
Y
y
Y
|||A||| A

y
X
y
Y
so A

y
X
|||A||| y
Y
and thus |||A

||| |||A||| and hence A

B(Y, X).
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Claim 3. A

= A.
Since we have shown A

B(Y, X), we can legitimately dene the adjoint of A

, denoted A

, as the (bounded linear) operator

that satises A

y, x
X
= y, A

x
Y
, x X, y Y.
Since y, Ax
Y
= Ax, y

Y
= x, A

X
= A

y, x
X
= y, A

x
Y
, by the previous uniqueness arguments we see A

= A.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Claim 4. |||A

||| = |||A|||.
From Claim 2 with A

: |||A

||| |||A

||| or equivalently: |||A||| |||A

|||. 2
Corollary. Under the same conditions as above: |||A

A||| = |||AA

||| = |||A|||
2
= |||A

|||
2
.
Proof. Recalling that |||S T ||| |||S|||||| T ||| we have by the preceding theorem:
|||A

A||| |||A

||||||A||| = |||A|||
2
= |||A

|||
2
,
and Ax
2
Y
= Ax, Ax
Y
= Ax, (A

x
Y
= A

Ax, x
X
A

Ax x |||A

A||| x
2
so |||A|||
2
|||A

A|||. Combining yields the equality |||A|||

2
= |||A

A|||. The rest is obvious using A

= A. 2
Proposition. If A B(X, Y) is invertible, where X,Y are Hilbert spaces, then A

has an inverse and (A

)
1
= (A
1
)

.
Proof.
Claim 1. A

: Y X is one-to-one into X.
Consider y
1
, y
2
Y with y
1
= y
2
. but suppose A

y
1
= A

y
2
, so A

d = 0 where d = y
2
y
1
= 0.
Thus x, A

d
X
= 0, x X and hence Ax, d
Y
= 0, x X.
Since A is invertible, we can make the change of variables z = Ax and hence z, d
Y
= 0, z X.
But this implies d = 0, contradicting the supposition that d = 0. So A

is one-to-one.
Claim 2. A

: Y X is onto X.
By the Banach inverse theorem, since A B(X, Y) and A is invertible, A
1
B(Y, X), so A
1
has its own adjoint, (A
1
)

.
Pick any z X. Then for any x X
x, z = A
1
Ax, z = Ax, (A
1
)

z = Ax, (A
1
)

z = x, A

(A
1
)

z .
Thus z = A

_
(A
1
)

z
. .
Y
_
, showing that z is in the range of A

. Since z X was arbitrary, A

is onto X.
Claim 3. (A

)
1
= (A
1
)

.
Since A

is one-to-one and onto X, A

is invertible.
Furthermore, z = A

(A
1
)

z =(A

)
1
z = (A
1
)

z, z X. Thus (A

)
1
= (A
1
)

. 2
Remark. AB = I and BA = I =A, B invertible and A
1
= B.
Remark. AB = I and A invertible =A
1
= B.
c J. Fessler, December 21, 2004, 13:3 (student version) 6.13
Denition. A B(H, H), where H is a real Hilbert space, is called self adjoint if A

= A.
Exercise. An orthogonal projection P
M
: H H, where M is a closed subspace in a Hilbert space H, is self adjoint. ??
Exercise. Conversely, if P B(H, H) and P
2
= P and P

= P, then P is an orthogonal projection operator. (L6.16)

Denition. A self-adjoint bounded linear operator A on a Hilbert space H is positive semidenite iff x, Ax 0, x H.
Remark. It is easily shown that x, Ax is real when A is self-adjoint.
Example. When M is a Chebyshev subspace in an inner product space, is P
M
a self-adjoint operator? ??
Unitary operators (Caution: the proof on [3, p. 358] is incomplete w.r.t. the onto aspects.)
We previously dened unitary operators; we now examine the adjoints of these.
Theorem. Suppose U B(X, Y) where X,Y are Hilbert spaces. Then the following are equivalent:
1. U is unitary, i.e., U is an isomorphism (linear, onto, and one-to-one) and Ux, Uz = x, z, x, z X,
2. U

U = I and UU

= I,
3. U is invertible with U
1
= U

.
Proof. (2 =3) and (3 =1) are obvious.
(1 =2) If U is unitary then for all x, z X: x, z
X
= Ux, Uz = U

Ux, z, so U

Ux = x, x X so U

U = I
X
.
For any y Y, since U is onto there exists an x X s.t. Ux = y. Thus UU

y = UU

Ux = UI
X
x = Ux = y.
Since y Y was arbitrary, UU

= I
Y
. 2
Remark. A corollary is that if U is unitary, then so is U

.
Remark. To see why we need both U

U = I and UU

= I above, consider U =
_
1
0
_
, for which U

U = I but U is not onto.

Example.
Consider X = Y =
2
and the (linear) discrete-time convolution operator A :
2

2
dened by
z = Ax z
n
=

k=
h
nk
x
k
, n Z,
where we assume that h
1
, which is equivalent to BIBO stability. We showed previously that Ax
2
h
1
x
2
, so A is
bounded with |||A||| h
1
, so A has an adjoint. (Later we will show |||A||| = H

where H is the frequency response of h.)

Since A is bounded, it is legitimate to search for its adjoint:
Ax, y =

n=
y

n
_

k=
x
k
h
nk
_
=

k=
x
k
_

n=
y
n
h

nk
_

k=
x
k
[A

y]

k
= x, A

y,
where the adjoint is
[A

y]
k
=

n=
h

nk
y
n
=[A

y]
n
=

k=
h

kn
y
k
,
which is convolution with
_
h

kn
_
.
When is A self adjoint? When h
l
= h

l
, i.e., h is Hermitian symmetric.
When is A unitary? When h
n
h

n
= [n], i.e., when |H()|
2
= 1.
6.14 c J. Fessler, December 21, 2004, 13:3 (student version)
6.6
Relations between the four spaces
The following theorem relates the null spaces and range spaces of a linear operator and its adjoint.
Remark. Luenberger uses the notation [R(A)] but this seems unnecessary since R(A) is a subspace.
Theorem. If A B(X, Y) where X,Y are Hilbert spaces, then
1. {R(A)}

= N(A

), 2. R(A) = {N(A

)}

,
3. {R(A

)}

= N(A), 4. R(A

) = {N(A)}

.
Proof.
Claim 1. {R(A)}

= N(A

)
Pick z N(A

) and any y R(A), so y = Ax for some x X.

Now y, z
Y
= Ax, z
Y
= x, A

z
X
= x, 0
X
= 0.
Thus z N(A

) =z {R(A)}

since y R(A) was arbitrary. So N(A

) {R(A)}

.
Now pick y {R(A)}

. Then for all x X, 0 = Ax, y

Y
= x, A

y
X
. So A

y = 0, i.e., y N(A

).
Since y {R(A)}

was arbitrary, {R(A)}

N(A

). Combining: {R(A)}

= N(A

)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Claim 2. R(A) = {N(A

)}

,
Taking the orthogonal complement of part 1 yields {R(A)}

= N(A

.
Recall from proposition 3.4-1 that S

= [S] when S is a subset in a Hilbert space. Since R(A) is a subspace, [R(A)] = R(A).
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Parts 3 and 4 follow by applying 1 and 2 to A

and using the fact that A

= A. 2
Example. (To illustrate why we need closure in R(A) above. Modied from [4, p. 156].)
Consider the linear operator A :
2

2
dened by Ax = (x
1
, x
2
/

2, x
3
/

3, . . .). Clearly x
2
= Ax
2
and
Ax x, so A is bounded (and hence continuous). In fact |||A||| = 1 (consider x = e
1
).
Clearly R(A) includes all nitely nonzero sequences, so R(A) =
2
.
However, y = (1, 1/2, 1/3, . . .) / R(A) (Why not?) yet y
2
, so R(A) is not closed.
This problem never arises in nite-dimensional spaces!
Example. Consider A :
2

2
dened by the upsampling operator: Ax = (x
1
, 0, x
2
, 0, x
3
, 0 . . .).
It is easily shown that A is bounded and |||A||| = 1.
One can verify that the adjoint operation is downsampling: A

y = (y
1
, y
3
, y
5
, . . .).
Clearly R(A

) =
2
and N(A) = {0} = [R(A

)]

.
Furthermore, R(A) = [

i=0
e
2i+1
] and N(A

) = [

i=1
e
2i
] and these two spaces are orthogonal complements of one another.
c J. Fessler, December 21, 2004, 13:3 (student version) 6.15
Example. Consider A : L
2
[R] L
2
[R] (which is complete [3, p. 589]) dened by the shifted ltering operator:
y = Ax y(t) = (Ax)(t) =
_

sinc
2
(t 3 ) x() d,
where sinc(t) =
sin(t)
t
or unity if t = 0. The adjoint is
(A

y)(t) =
_

sinc
2
(t + 3 ) y() d.
The nullspace of A

consists of signals in L
2
whose spectrum is zero over the frequencies (1/2, 1/2). The range of A is all
signals in L
2
that are band-limited to that same range, so the orthogonal complement is the same as N(A

). (Picture) .
Exercise. Why did I use sinc
2
() rather than sinc()? ??
6.7
Duality relations for convex cones skip
6.8
Geometric interpretation of adjoints (Presented in terms of general adjoints.) skip
When A B(X, Y) with X and Y Hilbert spaces, consider the following hyperplanes:
V
1
=
_
x X : x, A

y
X
= 1
_
for some y Y, V
2
=
_
y Y : Ax
0
, y
Y
= 1
_
for some x
0
X.
6.16 c J. Fessler, December 21, 2004, 13:3 (student version)
Optimization in Hilbert spaces
Consider the problem of solving y = Ax, where y Y is given, x X is unknown, A B(X, Y) is given, and X and Y are
Hilbert spaces. For any such y, there are three possibilities for x:
a unique solution,
no solution,
multiple solutions.
If A is invertible, then there is a unique solution x = A
1
y, which is the least interesting case.
6.9
The normal equations (No exact solutions, so we seek a minimum-norm, unconstrained approximation.)
We previously explored the normal equations in a setting where R(A) was nite dimensional. Now we have the tools to generalize.
The following theorem illustrates the fundamental role of adjoints in optimization.
Theorem. Let A B(X, Y) where X and Y are Hilbert spaces.
For a xed y Y, a vector x X minimizes y Ax
Y
iff A

Ax = A

y.
Proof. Consider the subspace M = R(A). Then the minimization problem is equivalent to inf
mM
y m.
By the pre-projection theorem, m

M achieves the inmum iff y m

M, i.e., y m

= [R(A)]

= N(A

), by a
previous theorem. Thus, 0 = A

(y m

) = A

y A

Ax, for some x X. 2

There is no claim of existence here, since R(A) might not be closed.
There is no claim of uniqueness of x

here, since although m

will be unique, there may be multiple solutions to m

= Ax.
If a minimum distance solution x

exists and A

A is invertible, then the solution is unique and has the familiar form:
x

= (A

A)
1
A

y.
Example. Find minimum-norm approximation to y H of the form y =

n
i=1

i
x
i
, where the x
i
s are linearly independent
vectors in H.
We know how to solve this from Ch. 3, but the operator notation provides a concise expression.
Dene the operator A B(C
n
, H) by
A
n

i=1

i
x
i
where = (
1
, . . . ,
n
).
Note that A =

G
_

max
(G) where G = A

A is the Gram matrix. Since the x

i
s are linearly independent, G
is symmetric positive denite so its eigenvalues are real and positive. So A is bounded and |||A||| =
_

max
(G).
Our goal is to minimize y A over C
n
. By the preceding theorem, the optimal solution must satisfy A

A = A

y.
What is A

: H C
n
here? Recall we need A, y
H
= , A

y
C
n, C
n
, so
A, y
H
=
_
n

i=1

i
x
i
, y
_
H
=
n

i=1

i
x
i
, y
H
=
n

i=1

i
y, x
i

H
=
n

i=1

i
[A

y]

i
= , A

y
C
n,
where we see that [A

y]
i
= y, x
i

H
and hence
A

y =
_
y, x
1

H
, . . . , y, x
n

H
_
.
Thus one can easily show that A

Ax = A

y is equivalent to the usual normal equations.

So no computational effort has been saved, but the notation is more concise. Furthermore, the notation (A

A)
1
A

y is comfort-
ingly similar to the notation (A

A)
1
A

y that we use for the least-squares solution of linear systems of equations in Euclidean
space. So with everything dened appropriately, the generalization to arbitrary Hilbert spaces is very natural.
c J. Fessler, December 21, 2004, 13:3 (student version) 6.17
6.10
The dual problem (for minimum norm solutions)
If y = Ax has multiple solutions, then in some contexts it is reasonable to choose the solution that minimizes some type of norm.
However, the appropriate norm is not necessarily the norm induced by the inner product.
Exercise. Generalize Luenbergers treatment to weighted norms.
Theorem. Let X and Y be Hilbert spaces and A B(X, Y). Suppose y R(A) is given, i.e., y = Ax
0
for some x
0
X.
Assume that R(A

) is closed in X. ERROR in Luenberger p. 161.

The unique vector x

X having minimum norm and satisfying Ax = y is characterized by:

x

= {A

z : AA

z = y, z Y} .
Proof. Since x
0
is one solution to Ax = y, the general solution has the form x = x
0
m, where m M N(A).
In other words, we seek the minimum norm vector in the linear variety
V = {x X : Ax = y} = {x
0
m: m M} .
Since A is continuous, N(A) is a closed subspace (homework). Thus V is a closed linear variety in a Hilbert space, and as such
has a unique element x

of minimum norm by the (generalized) projection theorem, and that element is characterized by the two
conditions x

= x
0
m

M, and x

V .
Since R(A

) was assumed closed, by the previous 4-space theorem we have M

= {N(A)}

= R(A

) = R(A

) .
Thus x

M =x

= R(A

) =x

= A

z for some z Y. (There may be more than one such z.)

Furthermore, x

V =Ax

= y =AA

z = y. (x

is unique even if there are many such z values!) 2

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
If AA

is invertible, then the minimum norm solution has the form:

x

= A

(AA

)
1
y.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Example. X = R
3
, Y = R
2
, A =
_
1 0 1
0 1 1
_
, y = (3, 3).
There are multiple solutions including x
1
= (0, 0, 3) and x
2
= (3, 3, 0) and convex combinations thereof.
Here A

= A
T
so AA

=
_
2 1
1 2
_
, [AA

]
1
=
1
3
_
2 1
1 2
_
, z = [AA

]
1
y =
_
1
1
_
and x

= A

z = (1, 1, 2).
Of course x

has a smaller 2-norm than the other solutions above.

However, x
1
is more sparse which can be important in some applications.
Example. X = R
1
, Y = R
2
, A =
_
1
1
_
, y = (2, 2). Then, using MATLABs pinv function, x = 2.
In this case, AA

=
_
1 1
1 1
_
, so there are multiple solutions to AA

z = y, each of which leads to the same x = A

z however!
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Example. The downsampling by averaging operator A :
2

2
is dened by Ax = (x
1
/2 +x
2
/2, x
3
/2 +x
4
/2, . . .).
One can show that this is bounded with |||A||| = 1/

2, since |||[1/2 1/2]||| = 1/

2.
The adjoint is A

y = (y
1
/2, y
1
/2, y
2
/2, y
2
/2, . . .), so AA

z = (z
1
/2, z
2
/2, . . .) =
1
2
z =AA

=
1
2
I. So z = 2y.
Thus x

= A

z = 2A

y = (y
1
, y
1
, y
2
, y
2
, . . .), which is a sensible solution in this application.
6.18 c J. Fessler, December 21, 2004, 13:3 (student version)
6.11
Pseudo-inverse operators
This concept allows a more general treatment of nding solutions to Ax = y, regardless of how many such solutions.
Denition. Let X and Y be Hilbert spaces with A B(X, Y) and R(A) closed in Y. For any y Y, dene the following linear
variety:
V
y
=
_
x
1
X : Ax
1
y
Y
= min
xX
Ax y
Y
_
.
Among all vectors x
1
V
y
, let x
0
be the unique vector of minimum norm
X
.
The pseudo-inverse A
+
of A is the operator mapping each y in Y into its corresponding x
0
. So A
+
: Y X.
Note: closure of R(A) usually arises from one of X or Y being nite dimensional.
This denition is legitimate since min
xX
y Ax = min
mM=R(A)
y m where R(A) is assumed closed.
By the projection theorem there is a unique y M = R(A) of minimum distance to y.
However, the linear variety V = {x X : y = Ax} may nevertheless contain multiple points.
What about uniqueness of the vector having minimum norm?
The set {x
1
X : Ax
1
= y} is a linear variety, a translate of N(A), which is closed. Why? ??
So by the Ch. 3 theorem on minimizing norms within a linear variety, x
0
is unique. Thus A
+
is well dened.
If A is invertible, then x
0
= A
1
y will of course be the minimizer, in which case we have A
+
= A
1
.
A
A
+
R(A)
N(A)
{N(A)}

x
x
0
y
y
V = {x X : Ax = y}
Often A
+
is many-to-one since many y vectors will map to the same x
0
.
Geometric interpretation (The above denition is algebraic.)
Since N(A) is a closed subspace in the Hilbert space X, by the theorem on orthogonal complements we have
X = N(A) {N(A)}

.
Similarly, since we have assumed that R(A) is closed (and it is a subspace):
Y = R(A) {R(A)}

.
When restricted to the subspace {N(A)}

, the operator A is a mapping from {N(A)}

to R(A) (of course).

Between these spaces, A is one-to-one, due to the following Lemma.
Thus A has a linear inverse on R(A) that maps each point in R(A) back into a point in {N(A)}

. This inverse denes A

+
on
R(A). To dene A
+
on all of Y, dene A
+
y = 0 for y {R(A)}

.
One way to write this is: A
+
=
_
A| {N(A)}

_
1
P
R(A)
.
Lemma. Let A : X Y be a linear operator on a Hilbert space X.
If S {N(A)}

, then A is one-to-one on S.
Proof. Suppose As
1
= As
2
. where s
1
, s
2
S {N(A)}

, which is a subspace, so s = s
1
s
2
{N(A)}

.
By the linearity of A, we have As = 0, so s N(A). Thus s = 0 and hence s
1
= s
2
. So A is one-to-one on S. 2
c J. Fessler, December 21, 2004, 13:3 (student version) 6.19
A
A A
+
A
+
0 0
R(A)
[R(A)]

[N(A)]

N(A)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Example. Consider A =
_
0 1
0 1
_
. Then
N(A) = {(a, 0) : a R} , {N(A)}

= {(0, b) : b R} ,
R(A) = {(b, b) : b R} , {R(A)}

= {(a, a) : a R} .
Recall that
y = P
[v]
x y =
_
x,
v
v
_
v
v
,
so P
R(A)
=
1
2
_
1
1
_
[1 1] =
1
2
_
1 1
1 1
_
.
And clearly
_
A| {N(A)}

_
1
=
_
0 0
0 1
_
, so A
+
=
_
A| {N(A)}

_
1
P
R(A)
=
_
0 0
0 1
_
1
2
_
1 1
1 1
_
=
_
0 0
1/2 1/2
_
.
Example.
(aI)
+
=
_
1
a
I, a = 0
0I, a = 0
= a
+
I.
Here are algebraic properties of a pseudoinverse.
Proposition. Let A B(X, Y) have closed range with pseudo-inverse A
+
: Y X. Then
A
+
B(Y, X)
(A
+
)
+
= A
A
+
AA
+
= A
+
AA
+
A = A
(A

)
+
= (A
+
)

(A
+
A)

= A
+
A = A

(A
+
)

A
+
= (A

A)
+
A

= A

(AA

)
+
A
+
= (A

A)
1
A

if A

A is invertible
A
+
= A

(AA

)
1
if AA

is invertible
In nite-dimensional spaces, a simple formula is given in terms of the SVD of A.
Proof. (Exercise)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
L6.19:
A
+
= lim
0
+
[A

A+ I]
1
A

= lim
0
+
A

[AA

+ I]
1
,
where the limits represent convergence with respect to what norm? ??
6.20 c J. Fessler, December 21, 2004, 13:3 (student version)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
One property that is missing is (CB)
+
= B
+
C
+
, unlike with adjoints and inverses.
However, L 6.21 claims that if B is onto and C is one-to-one, then (CB)
+
= B
+
C
+
.
Example.
C = [1 1], C

=
_
1
1
_
, C
+
= C

(CC

)
1
=
_
1/2
1/2
_
, B =
_
1
0
_
, B

= [1 0], B
+
= (B

B)
1
B

= [1 0].
In this case, CB = 1, but B
+
C
+
= 1/2.
Example. Something involving DTFT/ltering following DTFT analysis.
Example. Downsampling by averaging. (Handwritten notes.)
Example. (p.165)
From regularization design in tomography, form a LS approximation of the form: f()

3
k=0

k
cos
2
_


4
k
_
.
But those cos terms are linearly dependent!
More generally, if the x
i
s may be linearly dependent. how to we work with
min
C
n
_
_
_
_
_
y
n

i=1

i
x
i
_
_
_
_
_
.
Dene A : C
n
H by =

n
i=1

i
x
i
, where y, x
i
H.
If minimizing not unique, we could choose the one of minimum norm.
Here R(A) is closed since it is a nite-dimensional subspace of H.
So A
+
exists and A
+
= (A

A)
+
A

, where G = A

A is simply the n n Gram matrix. So the minimum norm LS solution is

= A
+
y = (A

A)
+
A

y.
Since Gis (Hermitian) symmetric nonnegative denite, it has an orthogonal eigenvector decomposition G = QDQ

, and one can

show that G
+
= QD
+
Q

.
c J. Fessler, December 21, 2004, 13:3 (student version) 6.21
Analysis of the DTFT
Given a discrete-time signal g(n), i.e., g : Z C, the discrete-time Fourier transform or DTFT is dened in introductory
signal processing books as follows:
G = Fg G() =

n=
g(n) e
n
. (6-2)
This is an innite series, so for a rigorous treatment we must nd suitable normed spaces in which we can establish convergence.
The natural family of norms is
p
, for some 1 p . Why? What about the doubly innite sum? ??
The logical meaning of the above denition is really
G = Fg G = lim
N
F
N
g, where G
N
= F
N
g G
N
()
N

n=N
g(n) e
n
, where N N. (6-3)
Alternatively, one might also try to show that F = lim
N
F
N
, where the limit is with respect to the operator norm in
B(
p
, L
r
[, ]) for some p and r, but this is in fact false! (See below.)
Since G
N
() is only a nite sum, clearly it is always well dened.
Furthermore, being a nite sum of complex exponentials, G
N
() is continuous in , and hence Lebesgue integrable on [, ].
So we could write F
N
: R
2N+1
L
1
[, ] or perhaps more usefully: F
N
:
p
L
r
[, ] for any 1 p, r .
To elaborate, note that by H olders inequality:
|G
N
()| =

n=N
g(n) e
n

n=N
|g(n)| =

n=
|g(n) 1
{|n|N}
| g
p
_
_
_ 1
{|n|N}
_
_
_
q
= g
p
(2N + 1)
11/p
.
Thus
F
N
g
r
= G
N

r
=
__

|G
N
()|
r
d
_
1/r
(2)
1/r
g
p
(2N + 1)
11/p
. (6-4)
Furthermore, for p = 1 the upper bound is achieved when g(n) = [n], so |||F
N
|||
1r
= (2)
1/r
.
Thus F
N
B(
p
, L
r
[, ]) for any 1 p, r , and |||F
N
|||
pr
(2)
1/r
(2N + 1)
11/p
.
Remark. f L

[a, b] =f L
r
[a, b] if < a < b < and r 1.
But to make (6-2) rigorous we must have normed spaces in which the limit in (6-3) exists.
Non-convergence of the operators
Note: treating F
N
:
p
L
r
[, ], by considering g
0
(n) = [n (M + 1)] we have for N > M:
|||F
N
F
M
|||
pr
= sup
g : g
p
1
(F
N
F
M
)g
r
(F
N
F
M
)g
0

r
=
_
_
_e
(M1)
0
_
_
_
r
= (2)
1/r
.
So {F
N
} is not Cauchy (and hence not convergent) in B(
p
, L
r
[, ]), no matter what p or r values one chooses.
So we must analyze convergence of the spectra G
N
() = F
N
g, rather than convergence of the operators F
N
themselves.
6.22 c J. Fessler, December 21, 2004, 13:3 (student version)

1
analysis
Proposition. If g
1
, then {F
N
g} is Cauchy in L
r
[, ] for any 1 r .
Proof.
If g
1
, then dening
I(N, M) {n Z : min(N, M) < |n| max(N, M)} (6-5)
and G
N
= F
N
g we have
|G
N
() G
M
()| =

n=N
g(n) e
n

n=M
g(n) e
n

nI(N,M)
g(n) e
n

nI(N,M)
|g(n)|

|n|>min(N,M)
|g(n)| 0 as N, M . (6-6)
So for each R, the sequence {G
N
()}

N=1
is Cauchy in R, and hence convergent by the completeness of R, provided g
1
.
Thus for each , {G
N
()} converges pointwise to some limit, call it G(), where (6-2) is shorthand for that limit.
Furthermore, when g
1
:
G
N
G
M

= sup
||
|G
N
() G
M
()| 0 as N, M .
So the sequence of function {G
N
} is Cauchy in L

[, ], which is complete, so {G
N
} converges to a limit G L

[, ].
More generally, using (6-6):
G
N
G
M

r
r
=
_

|G
N
() G
M
()|
r
d 2

|n|>min(N,M)
|g(n)| 0 as N, M ,
so {G
N
} is Cauchy in L
r
[, ] for any 1 r . 2
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Thus, due to completeness, {G
N
} converges to a limit G L
r
[, ].
So we can dene the DTFT operator F :
1
L
r
[, ] by
Fg lim
N
F
N
g. (6-7)
Proposition. F B(
1
, L
r
[, ]) for any 1 r with |||F|||
1r
= (2)
1/r
.
Proof. Linearity of F follows from linearity of F
N
. For g
1
:
Fg
r
=
_
_
_ lim
N
F
N
g
_
_
_
r
= lim
N
F
N
g
r
lim
N
|||F
N
|||
1r
g
1
= (2)
1/r
g
1
,
using (6-4). Equality is achieved when g(n) = [n]. 2
c J. Fessler, December 21, 2004, 13:3 (student version) 6.23

2
analysis
Unfortunately,
1
analysis is a bit restrictive; the class of signals is not as broad as we might like, and for least-squares problems
we would rather work in
2
. This will allow us to apply Hilbert space methods.
However, if a signal is in
2
, it is not necessarily in
1
, so the above
1
analysis does not apply to many signals in
2
. So we need a
different approach.
Proposition. If g
2
, then {F
N
g} is Cauchy in L
2
[, ].
Proof. If g
2
,then using G
N
= F
N
g:
G
N
G
M

2
2
=
_

n=N
g(n) e
n

n=M
g(n) e
n

2
d =
_

nI(N,M)
g(n) e
n

2
d
=

nI(N,M)

mI(N,M)
g(n)g

(m)
_

e
(nm)
d =

nI(N,M)

mI(N,M)
g(n)g

(m)2 1
{n=m}
= 2

nI(N,M)
|g(n)|
2
2

|n|>min(N,M)
|g(n)|
2
0 as N, M ,
since g
2
. Thus {F
N
g} is Cauchy in L
2
[, ]. 2
Since L
2
[, ] is complete, {G
N
} is convergent (in the L
2
sense!) to some limit G L
2
[, ], and the expression in (6-2) is
again a reasonable shorthand for that limit, whatever it may be, and now we can dene F :
2
L
2
[, ] via (6-7). In other
words,
G
N
G
M

2
2
=
_

|G
N
() G()|
2
d 0.
This is often called mean square convergence.
Proposition. F B(
2
, L
2
[, ]) with |||F|||
22
=

2.
Proof. Linearity of F is easily shown.
Since {G
N
} is convergent it is bounded. In fact
G
N

2
2
=
_

n=N
g(n) e
n

2
d =
N

n=N
N

m=N
g(n)g

(m)
_

e
(nm)
d = 2
N

n=N
|g(n)|
2
2 g
2
2
,
so Fg
2

2 g
2
and hence |||F|||

2.
Furthermore, if we consider g(n) = [n], then G() = 1.
Thus G
2
2
=
_

1 d = 2 which achieves the upper bound above. Hence |||F|||

22
=

2. 2
6.24 c J. Fessler, December 21, 2004, 13:3 (student version)
Adjoint
Since F B(
2
, L
2
[, ]) and both
2
and L
2
[, ] are Hilbert spaces, F has an adjoint:
Fg, S
L2[,]
=
_

()
_

n
g(n) e
n
_
d =

n
g(n)
__

S() e
n
d
_

= g, F

S
2
where x = F

S x(n) =
_

S() e
n
d, n Z.
So the adjoint is almost the same as the inverse DTFT (dened below).
And of course we know that F

B(L
2
[, ],
2
).
Range
To apply the Hilbert space methods, we would like R(F) to be closed in L
2
[, ]. It sufces for R(F) to be onto L
2
[, ],
since of course L
2
[, ] itself is closed.
Proposition. The DTFT F :
2
L
2
[, ] dened by (6-2) is onto L
2
[, ].
Proof. Let e
k
, k Z denote the family of functions e
k
() =
1

2
e
k
.
Recall that {e
k
} is a complete orthonormal basis for L
2
[, ] [4, p. 62].
Thus, by Parsevals relation we have for any G L
2
[, ]:
G =

k
G, e
k
e
k
, i.e., G() =

k
G, e
k

2
e
k
,
where

k
| G, e
k
|
2
= G
2
2
.
Thus if we dene g(k) = G, e
k
/

2 then g
2
and G = Fg, so G R(F).
Since G L
2
[, ] was arbitrary, we conclude R(F) = L
2
[, ]. 2
DTFT is (almost) unitary
One can show easily that
Fu, Fv
L2[,]
= 2 u, v
2
.
Thus (since F is linear and invertible and hence an isomorphism), the normalized DTFT U = 1/

2F is unitary.
So L
2
[, ] and
2
are unitarily equivalent.
c J. Fessler, December 21, 2004, 13:3 (student version) 6.25
Inverse DTFT
Dene a partial inverse DTFT operator R
N
: L
2
[, ]
2
by
g
N
= R
N
G g
N
(n) =
1
2
_

G() e
n
d 1
{|n|N}
=
1

2
G, e
n
1
{|n|N}
.
Proposition. If G L
2
[, ], then {R
N
G} is Cauchy in
2
.
Proof.
R
N
GR
M
G =
1
2

kI(N,M)
| G, e
k
|
2

1
2

|k|>min(N,M)
| G, e
k
|
2
0 as N, M .
2
Since
2
is complete, {R
N
G} converges to some limit g
2
and we dene RG to be that limit: RG lim
N
R
N
G.
Proposition. R B(L
2
[, ],
2
) and |||R||| = 1/

2.
Proof.
R
N
G
2
2
=

|n|N

1
2
_

G() e
n
d

2
=
1
2

|k|N
| G, e
k
|
2

1
2

k
| G, e
k
|
2
=
1
2
G
2
2
.
So |||R
N
||| 1/

2 and R
N
B(L
2
[, ],
2
).
When G() = 1 we have R
N
G
2
= [n]
2
= 1 and G
2
= 2 so |||R
N
||| = 1/

2.
Proof 2. RG
2
= lim
N
R
N
G
2
lim
N
1

2
G
2
. Consider G = 1 to show equality. 2
Proposition. RF = I
2
and FR = I
L2
, so F
1
= R, where I
H
denotes the identity operator for Hilbert space H.
Proof. Exercise. 2
6.26 c J. Fessler, December 21, 2004, 13:3 (student version)
Convolution revisited
Using time-domain analysis, we showed previously that if h
1
and Ax = h x then A B(
p
,
p
).
We have shown F B(
2
, L
2
[, ]) and F
1
B(L
2
[, ],
2
).
Consider the band-limiting linear operator D : L
2
[, ] L
2
[, ] dened by
y = Dx y() =
_
x(), || /2
0, otherwise.
Clearly D B(L
2
[, ], L
2
[, ]) and in fact |||D||| = 1.
Now consider A F
1
DF. We previously showed in the analysis of the composition of operators that |||ST||| |||S||||||T|||.
So A B(
2
,
2
) with |||A||| |||F
1
||||||D||||||F|||
1

2
1

2 = 1.
But this A represents an ideal lowpass lter, i.e., convolution with h(n) =
1
2
sinc
_
n
2
_
. But this h /
1
.
Evidently, the convolution operator, at least in
2
, has a looser requirement than h
1
.
In contrast, in

, h
1
is both necessary and sufcient for A
h
B(

).
In
2
, a necessary and sufcient condition is that the frequency response be bounded.
Proposition. A
h
B(
2
,
2
) (with |||A
h
||| = H

) Fh

< , i.e., Fh L

[, ] .
Proof. (=) Suppose Fh

is nite. Then using the convolution property of the DTFT and Parseval:
h x
2
2
=
1
2
HX
2
2
=
1
2
_

|H()X()|
2
d H
2

1
2
X
2
2
= H
2

x
2
2
,
|||A
h
||| Fh

= H

. The upper bounded is achieved when h(n) = [n].

(=) If H is unbounded, then forall T there exists an interval over which H T. Choose x to be a signal whose spectrum is an
indicator function on that interval, and then x h
2
T x
2
, so A
h
would be be unbounded. Take contrapositive. 2
Continuous-time case
Fourier transform
convolution
Youngs inequality
c J. Fessler, December 21, 2004, 13:3 (student version)
1. P. Eno. A counterexample to the approximation problem in Banach spaces. Acta Math, 130:30917, 1973.
2. I. J. Maddox. Elements of functional analysis. Cambridge, 2 edition, 1988.
3. A. W. Naylor and G. R. Sell. Linear operator theory in engineering and science. Springer-Verlag, New York, 2 edition, 1982.
4. D. G. Luenberger. Optimization by vector space methods. Wiley, New York, 1969.
5. J. Schauder. Zur theorie stetiger abbildungen in funktionenrumen. Math. Zeitsch., 26:4765, 1927.
6. L. Grafakos. Classical and modern Fourier analysis. Pearson, NJ, 2004.
7. P. P. Vaidyanathan. Generalizations of the sampling theorem: Seven decades after Nyquist. IEEE Tr. Circ. Sys. I, Fundamental
theory and applications, 48(9):1094109, September 2001.
8. A. M. Ostrowski. Solution of equations in Euclidian and Banach spaces. Academic, 3 edition, 1973.
9. R. R. Meyer. Sufcient conditions for the convergence of monotonic mathematical programming algorithms. J. Comput.
System. Sci., 12(1):10821, 1976.
10. M. Rosenlicht. Introduction to analysis. Dover, New York, 1985.
11. A. R. De Pierro. On the relation between the ISRA and the EM algorithm for positron emission tomography. IEEE Tr. Med.
Imag., 12(2):32833, June 1993.
12. A. R. De Pierro. On the convergence of the iterative image space reconstruction algorithm for volume ECT. IEEE Tr. Med.
Imag., 6(2):174175, June 1987.
13. A. R. De Pierro. Unied approach to regularized maximum likelihood estimation in computed tomography. In Proc. SPIE
3171, Comp. Exper. and Num. Meth. for Solving Ill-Posed Inv. Imaging Problems: Med. and Nonmed. Appl., pages 21823,
1997.
14. J. A. Fessler. Grouped coordinate descent algorithms for robust edge-preserving image restoration. In Proc. SPIE 3170, Im.
Recon. and Restor. II, pages 18494, 1997.
15. A. R. De Pierro. A modied expectation maximization algorithm for penalized likelihood estimation in emission tomography.
IEEE Tr. Med. Imag., 14(1):132137, March 1995.
16. J. A. Fessler and A. O. Hero. Penalized maximum-likelihood image reconstruction using space-alternating generalized EM
algorithms. IEEE Tr. Im. Proc., 4(10):141729, October 1995.
17. M. W. Jacobson and J. A. Fessler. Properties of MM algorithms on convex feasible sets. SIAM J. Optim., 2003. Submitted. #
061996.
18. P. L. Combettes and H. J. Trussell. Method of successive projections for nding a common point of sets in metric spaces. J.
Optim. Theory Appl., 67(3):487507, December 1990.
19. F. Deutsch. The convexity of Chebyshev sets in Hilbert space. In A. Yanushauskas Th. M. Rassias, H. M. Srivastava, editor,
Topics in polynomials of one and several variables and their applications, pages 14350. World Sci. Publishing, River Edge,
NJ, 1993.
20. M. Jiang. On Johnsons example of a nonconvex Chebyshev set. J. Approx. Theory, 74(2):1528, August 1993.
21. V. S. Balaganskii and L. P. Vlasov. The problem of convexity of Chebyshev sets. Russian Mathematical Surveys, 51(6):1127
90, November 1996.
22. V. Kanellopoulos. On the convexity of the weakly compact Chebyshev sets in Banach spaces. Israel Journal of Mathematics,
117:619, 2000.
23. A. R. Alimov. On the structure of the complements of Chebyshev sets. Functional Analysis and Its Applications, 35(3):176
82, July 2001.
24. Y. Bresler, S. Basu, and C. Couvreur. Hilbert spaces and least squares methods for signal processing, 2000. Draft.
25. M. Vetterli and J. Kovacevic. Wavelets and subband coding. Prentice-Hall, New York, 1995.
26. D. C. Youla and H. Webb. Image restoration by the method of convex projections: Part ITheory. IEEE Tr. Med. Imag.,
1(2):8194, October 1982.
27. M. Unser and T. Blu. Generalized smoothing splines and the optimal discretization of the wiener lter. IEEE Tr. Sig. Proc.,
2004. in press.