Regularization

1
Tikonov’s regularization theory

Example: least square estimator

Other examples

2
 Should depend on geometric properties of the
approximating function

Say,

Linear differential operator (prior)

Mapping from RKHS to another Hilbert space

We require f(x) to be a member of RKHS with reproducing

3 kernel in the form of Dirac delta distribution.
 Assuming least squares

Maps functions to real lines.

Sol:

Solution completely determined by examples.

Solution completely determined by prior,
examples unreliable.

4
Space
A set satisfying certain axioms

Metric Space
(𝑋𝑋, 𝑑𝑑)
A set with a metric defined on its elements

5
 Complete Metric Space
Every Cauchy sequence in the underlying set converges
to an element in the set

Limit point
𝑑𝑑 𝑥𝑥𝑚𝑚 , 𝑥𝑥𝑛𝑛 < 𝜀𝜀, ∀𝜀𝜀 > 0, 𝑚𝑚, 𝑛𝑛 > 𝑁𝑁
𝑑𝑑 𝑥𝑥𝑛𝑛 , 𝑥𝑥 → 0 𝑛𝑛 → ∞

Vector Space
A set with vectors as elements, and vector addition and
scalar multiplication on vectors are defined.
- Linear independence of vectors

6
 Dimension of a vector space

A vector space has ‘n’ dimensions if no more than n

element vectors for a linearly independent set

This subset is called Hamel basis of the vector space

Normed vector space (𝑋𝑋, . )

A vector space with a norm (vector length) defined on it

7
 A complete normed vector space is Banach space
Norm induced metric: 𝑑𝑑 𝑥𝑥, 𝑦𝑦 = 𝑥𝑥 − 𝑦𝑦

Inner product vector space (𝑋𝑋, <. , . >)

A vector space with an inner product defined on it

A complete inner product vector space is Hilbert space

Inner product induced norm & metric: < 𝑥𝑥, 𝑥𝑥 >, < 𝑥𝑥 − 𝑦𝑦, 𝑥𝑥 − 𝑦𝑦 >
8
 • Vector orthogonality (orthonormality) – inner product 0
• An orthonormal set (elements pairwise orthonormal) is
linearly independent set

Not all normed space are inner product spaces, must satisfy

Example- Euclidean norm

Hilbert space examples:

Euclidean space RN, CN

Sequence (infinite) space l2
Function space L2[a,b]

9
More.. Hilbert Space
Vector space with inner product structure defined

A vector

if

10
 Orthonormal

Cauchy!

Square
summability
11
Reproducing Kernel Hilbert Space

Kernel:

12
Mercer’s theorem:

Eigen
decomposition

Positive semi-
definite

13
Mercer’s theorem (repeat):

14
 If

Inner product / Mercer kernel

Vector space of real valued functions of x

generated by k

15
 Two of them

Kernel matrix

16
17
 Reproducing property

Reproducing kernel of

If completed:
Reproducing kernel Hilbert space.

18
 Uniqueness

RKHS Kernel

19