Image Restoration

(Sections 5.1, 5.5, 5.7, 5.9)

CS474/674 Prof. Bebis

Image Restoration
Image Enhancement
A subjective process using mostly heuristics (i.e., takes advantage
of the psychophysical aspects of the human visual system)
Image Restoration
An objective process assuming a-priori knowledge of the
degradation process.
blur due to motion
Modeling Image Degradation
Degradation process can be modeled through a degradation
function H and additive noise (x,y).


Goal of Image Restoration

Given some knowledge of H and noise (x,y), the
objective of restoration is to obtain an estimate of the
original image.
Assumptions about degradation function H
H is linear:




H is shift invariant:


(i.e., shifting the input merely shifts the output by the same amount)
Degradation Model
(assuming linearity and shift invariance)




Then, H[f(x,y)] is:

f(,b)H[(x-,y-b)ddb

Any function f(x,y) can be written as follows:

Degradation Model - Continuous Case (contd)
Since H(x,y) is shift invariant:



Therefore
impulse response
if
Degradation Model - Continuous Case (contd)
Under the assumptions of linearity and shift invariance:


( , ) ( , ) ( , ) ( , ) G u v H u v F u v N u v = +
or
Noise Properties
Noise arises typically during image acquisition and /or
transmission.
For simplicity, we will assume that:
(1) noise is independent of spatial coordinates
(2) there is no correlation between pixel values and values of
noise components

(* not true for periodic noise)


Noise Models
Most types of noise are modeled
using a probability density
function.

Noise model is typically chosen
based on some understanding
of the noise source.


salt & pepper
Gaussian noise
2 2
2 / ) (
2
1
) (
o
o t
z z
e z p

=
Gaussian noise arises in an
image due to factors such as:

(1) electronic circuit noise

(2) sensor noise due to poor
illumination and/or high
temperature

Rayleigh noise
4
) 4 (
4 /
0
) (
2
) (
2
/ ) (
2
t
o t

= + =

<
>
=

b
b a z
a z
a z e a z
b
z p
b a z

Typically used to characterize
noise phenomena in range
imaging.

Gamma (Erlang) noise
2
2
1
0 0
0
)! 1 (
) (
a
b
a
b
z
z
z e
b
z a
z p
az
b b
= =

<
>

o
Typically used to characterize
noise phenomena in laser
imaging.

Exponential noise
2
2
1 1
0 0
0
) (
a a
z
z
z ae
z p
az
= =

<
>
=

o
Typically used to characterize
noise phenomena in laser
imaging.

Uniform noise
12
) (
2
0
1
) (
2
2
a b b a
z
otherwise
b z a
a b
z p

=
+
=

s s

=
o
Least used in practice.

Useful as the basis for random
number generators.

Impulse (salt and pepper) noise

=
=
=
otherwise
b z P
a z P
z p
b
a
0
) (
P
a
=P
b
, a=0, and b=255, then
we salt and pepper noise.

Common in situations where
quick transients (e.g., faulty
switching), takes place during
imaging.

Noise Models - Example
test pattern
noise corrupted images and their histograms
Noise Models - Example
test pattern
noise corrupted images and their histograms
Estimation of noise parameters
Estimate the parameters of the noise PDFs from small
patches of reasonably constant background intensity.
For impulse noise, estimate probability of black/white
pixels (i.e., choose a midgray region).
(i) Estimate mean
and variance

(ii) Compute a and b
(i.e., parameters
of noise distributions)
Restoration in the presence of noise only
Restoration in the presence of noise only
(contd)
If noise can not be estimated, filtering methods can be used to
suppress noise:
Mean filters (arithmetic, geometric, harmonic etc.)
Order statistic filters (median, max and min, midpoint)
Frequency-domain filters (band-reject, band-pass, etc. )

Arithmetic/Geometric mean filters
,
( , )
1

( , ) ( , )
x y
s t S
f x y g s t
mn
e
=

,
1
( , )

( , ) ( , )
x y
mn
s t S
f x y g s t
e
(
=
(
(

[
m x n
mask
geometric mean
filters tend to
preserve more
details (i.e., less
blurring)
Contra-Harmonic filters
| |
| |
,
,
1
( , )
( , )
( , )

( , )
( , )
x y
x y
Q
s t S
Q
s t S
g s t
f x y
g s t
+
e
e
=

good for removing

salt or pepper
noise (but not both
simultaneously)

Q>0 pepper noise
Q<0 salt noise

Q: order of filter
If Q=0, same as arithmetic filter!
Contra-Harmonic filters (contd)
selecting the wrong sign in a contra-harmonic filter:
Order statistics filters
Order statistics filters are nonlinear spatial filters.

Their response is based on the ordering (ranking) the
pixels contained in an area covered by the filter.


Median filter
{ }
,
( , )

( , ) ( , )
x y
s t S
f x y median g s t
e
=
multiple passes improve results
less blurring compared
to arithmetic filters
Max/Min filters
Max filter is good
for pepper noise
Min filter is good
for salt noise
)} , ( { min ) , (

) , (
t s g y x f
xy
S t s e
=
)} , ( { max ) , (

) , (
t s g y x f
xy
S t s e
=
Alpha-trimmed mean filter
Good for multiple types
of noise (e.g., Gaussian noise
and salt-and-pepper noise)

Assume an m x n neighborhood:
(1) Disregard d/2 lowest
and d/2 highest values
(2) Average the remaining
values

Special cases:

d=0 arithmetic mean
d=mn-1 median
Adaptive Filters
The behavior of adaptive filters depends on the
statistical characteristics of the pixel values inside the
filtered region.

Typically, adaptive filters have superior performance
compared to non-adaptive filters.

But they have higher complexity too.
Adaptive median filtering
Size of filtered region is not fixed (see book for
details).

Periodic noise
Arises from electrical or electromechanical
interference during image acquisition.
Can be analyzed and filtered quite effectively in the
frequency domain.
Use a band-reject filter to isolate the noise:
image corrupted by sinusoidal noise spectrum of noisy image
Band-reject filters
Ideal Butterworth Gaussian
Band-reject filters (contd)
Bandpass filters
Performs the opposite operation of a band-reject filter.
H
BP
(u,v) = 1 - H
BR
(u,v)
Useful in isolating the effect of specific frequencies in
an image.
noise
pattern
Estimating degradation H
Typically, H is modeled mathematically, e.g.:
Atmospheric turbulance
Motion


G(u,v)=H(u,v)F(u,v) + N(u,v)
Degradation due to environmental conditions
Atmospheric turbulence (Hufnagel and Stanley [1964])
k=0.0025
k=0.001 k=0.00025
6 / 5 2 2
) (
) , (
v u k
e v u H
+
=
G(u,v)=H(u,v)F(u,v) + N(u,v)
Degradation due to uniform linear motion
Consider the case of object planar motion (i.e., 2D)
where x
0
(t) and y
0
(t) are the motion trajectories.




(x
0
(t), y
0
(t))
t=0
t=T
e.g., camera motion
Degradation due to uniform linear motion
If T is the exposure duration (i.e., time interval during
which the camera shutter is open), then the output
image g(x,y) is:



Taking the Fourier Transform:


}
=
T
dt t y y t x x f y x g
0
0 0
)) ( ), ( ( ) , (
dt e v u H
T
t vy t ux j
}
+
=
0
)) ( ) ( ( 2
0 0
) , (
t
G(u,v)=H(u,v)F(u,v)
where
(see book for
proof)
Degradation due to uniform linear motion
(contd)
If x
0
(t)=t/T and y
0
(t)=0, then:




If x
0
(t)=t/T and y
0
(t)=bt/T, then:




o t o t t
o t
o t
u j
T
T t u j
T
t ux j
e u
u
T
dt e dt e v u H

= = =
} }
) sin( ) , (
0
/ 2
0
)) ( ( 2
0
) (
)) ( sin(
) (
) , (
vb ua j
e vb ua
vb u
T
v u H
+
+
+
=
t
t
o t
Degradation due to uniform linear motion
(contd)
G(u,v)=H(u,v)F(u,v) where:
) (
)) ( sin(
) (
) , (
vb ua j
e vb ua
vb u
T
v u H
+
+
+
=
t
t
o t
Inverse Filtering
Inverse Filtering (contd)
Degradation model:



Idea: given H(u,v), estimate the original image as
follows:

( , ) ( , ) ( , ) ( , ) G u v H u v F u v N u v = +

( , ) ( , ) / ( , ) (1/ ( , )) ( , )
( , )
( , )
( , )
F u v G u v H u v H u v G u v
N u v
F u v
H u v
= =
= +
Inverse Filtering: Practical Issues

( , ) ( , ) / ( , )
( , )
( , )
( , )
F u v G u v H u v
N u v
F u v
H u v
=
= +
( , ) ( , ) ( , ) ( , ) G u v H u v F u v N u v = +
Inverse Filtering Practical Issues
If H(u,v) is zero or very small, then N(u,v)/H(u,v)
will dominate the estimate:




In practice, H(u,v)
and N(u,v) look like:

Inverse Filtering Practical Issues (contd)
H(u,v)
Restrict filtering to values
inside the circle only!
Solution: carry out restoration
in a limited neighborhood about
the origin to avoid small values
of H(u,v).

Inverse Filtering Practical Issues (contd)
full filter
H cutoff
radius 40
H cutoff
radius 70
H cutoff
radius 85
k=0.0025
inverse
filtering
6 / 5 2 2
) ) 2 / ( ) 2 / ((
) , (
N v M u k
e v u H
+
=
Inverse Filtering
Inverse filtering does not handle noise explicitly.



Wiener filtering integrates statistical properties of
noise into the restoration process.

( , ) ( , ) / ( , ) (1/ ( , )) ( , )
( , )
( , )
( , )
F u v G u v H u v H u v G u v
N u v
F u v
H u v
= =
= +
Minimum Mean Square Error Filtering
(Wiener Filtering)
Assume f (x,y) and q(x,y) are uncorrelated (plus a few
other assumptions, see textbook)


Objective is to minimize:


Also called least-square error filter


{ }
2

E f f
Minimum Mean Square Error Filtering
(Wiener Filtering) (contd)


2 2
| ) , ( | ) , ( | ) , ( | ) , ( v u F v u S v u N v u S
f
= =
q
(noise power spectrum) (image power spectrum)
2
2
1 | ( , ) |
( , ) ( , )
( , ) | ( , ) | ( , ) / ( , )
f
H u v
F u v G u v
H u v H u v S u v S u v
q
(
=
(
+
(

Special Case: S
n
(u,v) = 0
What happens if S
n
(u,v) = 0?

( , ) (1/ ( , )) ( , )
( , )
( , )
( , )
F u v H u v G u v
N u v
F u v
H u v
=
= +
Inverse
Filtering!
2
2
1 | ( , ) |
( , ) ( , )
( , ) | ( , ) | ( , ) / ( , )
f
H u v
F u v G u v
H u v H u v S u v S u v
q
(
=
(
+
(

Practical Issues
S
f
(u,v) and maybe S
n
(u,v) are unknown.
2
2
( , )
( , )

( , )
( , )
( , )
H u v
G u v
F u v
H u v
H u v K
=
+
2
2
( , )
( , )

( , )
( , )
( , ) ( , ) / ( , )
f
H u v
G u v
F u v
H u v
H u v S u v S u v
q
=
+
approximate S
n
(u,v) / S
f
(u,v) by K:
Wiener Filtering - Example
K was chosen manually
for best results
Wiener Filtering Example (contd)
motion blur
and additive
Gaussian noise
inverse filtering Wiener filtering
one order of
magnitude less noise
five orders of
magnitude less noise
inverse filtering
inverse filtering
Wiener filtering
Wiener filtering
K was chosen manually
for best results
Measures
How do we characterize the performance of different
image restoration algorithms?

2
1 1
0 0
1 1
2
0 0
1 1
2
0 0
Mean Square Error (MSE)
1
MSE= ( , ) ( , )
( )
( , )
SNR
[ ( , ) ( , )]
M N
x y
M N
u v
M N
u v
f x y f x y
MN
Signal to Noise Ratio SNR
f x y
f x y f x y

= =

= =

= =
(

Constrained Least Squares Filtering

Wiener filter
Requires knowledge of S
n
(u,v) and S
f
(u,v)
Optimal on average

CLS filter
Requires knowledge of the mean and variance of noise only.
Optimal for each image.
Warning: optimum restoration using CLS does not necessarily imply
best in the visual sense.

Expressing degradation
using matrix notation


f(x,y) is defined for
h(x,y) is defined for
Form the extended sequences
x=0,1,2,...,M-1
y=0,1,2,...,N-1
Expressing degradation
using matrix notation (contd)
.
.
.
x=0
x=1
x=M-1
x=0,1,2,...M-1 1D case:
Degradation using matrix notation (contd)
Using matrix notation
or
g: Mx1 H: M x M f: M x 1 n: M x 1
Degradation using matrix notation (contd)
2D case: assume an M x N image:
g: MNx1 H: MN x MN f: MN x 1 n: MN x 1
Matrix manipulations could be used but
- Very high dimensional vectors
- Need to invert H
- Solution is sensitive to noise
Constrained Least Squares (CLS) Filtering
Deal with sensitivity to noise using a measure of
smoothness:
Minimize second derivative (i.e., Laplacian)

Subject to:
(since )
where:
Constrained Least Squares Filtering (contd)
The solution is:



P(u, v) is the Fourier transform of the function:



is the only parameter (if = 0 inverse filtering)

(Laplacian)
Constrained Least Squares Filtering - Example
motion blur
and additive
Gaussian noise
inverse filtering Wiener filtering
one order of
magnitude less noise
five orders of
magnitude less noise
inverse filtering
inverse filtering
Wiener filtering
Wiener filtering
was chosen
manually
for best results
CLS
CLS
CLS
Determine iteratively

Degradation model:

Define residual error:

Adjust such that:
f H g r

=
o q s | || || || || |
2 2
r
2
where:
Note: ||r||
2
is a monotonically increasing function of
Determine iteratively (contd)
o q s | || || || || |
2 2
r
d increment :
f H g r

=
CLNS requires knowledge of noise mean and
variance ONLY!
We need to estimate

2
|| || q
where:
(mean of noise)
(variance of noise)
It can be shown that:
Constrained Least Squares Filtering - Example
k=0.0025
5
0
6
10 , 0.25
10 d
o

= =
=
Wiener output
k manually selected
CLS
correct noise parameters wrong noise parameters
Geometric Mean Filter
1
2
2 2
*( , ) | ( , ) |
( , ) ( , )
| ( , ) | | ( , ) | ( , ) / ( , )
f
H u v H u v
F u v G u v
H u v H u v S u v S u v
o
o
q
|

(
(
= (
(
( +
(


1: inverse filter
=0: parametric Wiener filter
=1/2: geometric mean filter
o
o
o
=