Image Warping

Transformation 2
Instructor: Mohsen Ali

Whats on the menu today?

Revision: Transformations
Recovering parameters
Assignment

Slides have been modified from Dr.Sohaib and Aamer Zaheers

lectures.

2-D Transformations
Scaling
Original

Translation

Shear

Affine

Rotation

Projective

Basic 2D Transformations
cos
sin

1 0 d x
0 1 d
y

0 0 1

sin
cos
0

0
0
1

1 ex
0 1

0 0

sx
0

0
0
0
1

0
sy
0
1
e
y
0

0
0
1
0 0
1 0
0 1

Rotation about an Arbitrary Point

Order of Transformations

Scaled

Rotated

Rotated

Scaled

Original

Original

Hierarchy of 2D Transformations

Length Ratios

Length Ratios
Length Cross-Ratios

Recovering
Best Affine Transformation
Input: we are given some correspondences
Output: Compute a1 a6 which relate the images

Recovering
Affine
Transformation
Input: weBest
are given
some correspondences
Output: Compute a1 a6 which relate the images

a1

a2

a3

a4

a5

a6

a7

a8

a9

This is an optimization problem Find the best set of parameters, given the input data

Recovering
Affine
Transformation
Input: weBest
are given
some correspondences
Output: Compute a1 a6 which relate the images

a1

a2

a3

a4

a5

a6

This is an optimization problem Find the best set of parameters, given the input data

Six Unknowns

Least Squares Error Solution

E (a) = ((a x + a y + a x ) + (a x
n

' 2
j

' 2
+
a
y
+
a

y
j
j)
5 j
6

j =1

Minimize E w.r.t. a
E
Compute a , put equal to zero, solve
simultaneously
i

x 2j
j
x y
j j

xj

j
0

x y x
y y
y 1
j

2
j

x x y
x y y
x
y
2
j

2
j

x j x 'j
0
j

'

0
yjxj
1

j
a
2
0 x 'j
a3
= j
'
x
x
y
a
j j 4 j j j

a
'
5
y
y
y
j j a j j j

6
'

j 1
j y j

Pseudo inverse
For an over-constrained linear system
Ax = b
A has more rows than columns
Multiply by AT on both sides
ATAx = ATb
ATA is a square matrix of as many rows as x
We can take its inverse
x = (ATA)-1ATb

x1
0

x2

0
x3

0
x
4
0

y1
0
y2
0

1 0
0 x1
1 0
0 x2

0
y1
0
y2

y3
0
y4
0

1 0
0 x3
1 0
0 x4

0
y3
0
y4

0
x1 '
1 a1 y1 '
0 a2 x2 '

1 a3 y2 '
=
0 a4 x3 '

1 a5
y3 '

0 a6 x4 '

1
y4 '
x

2D Affine Warping

Warping
Inputs:
Image X
Affine Transformation A = [a1 a2 b1 a3 a4 b2]T

Output:
Generate X such that X = AX

Obvious Process:
For each pixel in X
Apply transformation
At that location in X, put the same color as at the original location in X

Problems?

Warping

This will leave holes

Because every pixel does not map to an integer location!

Reverse Transformation

For each integer location in X

Apply inverse mapping
Problem?

Will not result in answers at integer locations, in general

Solution: Bilinearly interpolate from 4 neighbors

Interpolation
In 1D
3

Use y=mx+c
m: slope of the line
c: intersect
m = 1, c = -2

4
4.3 ?

Substitute x=4.3, => y = 2.3

2D Bilinear Interpolation
Four nearest points of (x, y)

(x, y )

(x, y )

(x, y ), (x, y ), (x, y ), (x, y )

( x, y )

where

x = int( x)
y = int( y )
x = x +1
y = y +1

(x, y )

(x, y )

Bilinear Interpolation
f ' ( x, y ) = x y f ( x, y ) + x y f ( x, y ) + x y f ( x, y ) + x y f ( x, y )

x = x x
y = y y
x = x x
y = y y

(x, y )

(x, y )
x

(x, y )

(x, y )

Warping
To apply Translation T of [0,1]
Calculate inv(T)

Translation T [0, 1]

(0,0)

(0,1)

(0,0)

(0,1)

(1,0)

(1,1)

(1,0)

(1,1)

I1

I2

For every coordinate (y,x) in I2

Apply inv(T)
E.g.
(0,0) -
0, -1 :::::: clip it
(0,1) -
0, 0 :::::: use value in (0,0)

Warping
What if the Translation is [0, 0.7]
Translation T [0, 0.7]
To apply Translation T of [0,0.7]
Calculate inv(T), [0, -0.7]

(0,0)

(0,1)

(0,0)

(0,1)

(1,0)

(1,1)

(1,0)

(1,1)

For every coordinate (y,x) in I2

Apply inv(T)
E.g.
(0,0) -
0, -0.7 :::::: clip it
(0,1) -
0, 0 .3 ::::: Where will you pick
value for I2( 0,1)?
Use Bilinear Interpolation

I1

I2

Warping Color Images

Same as applying to Gray Scale Images
Since Color images consists of
More than one 2D matrix
In case or RGB image
Apply transformation to each layer separately

Bilinear Interpolation
Also applied separately

3D Transformations

3D Translation
Point in 3D given by (X1 Y1 Z1)
Translated by (dx dy dz)
X2 = X1 + dx
Y2 = Y1 + dy
Z2 = Z1 + dz

Translation
In matrix form

X 2 1
Y 0
2=
Z 2 0

1 0

0 0 dx X 1
1 0 dy Y1
0 1 dz Z1

0 0 1 1
T

Inverse Transformation
1
0
T=
0

0 0 dx
1 0 dy
0 1 dz

0 0 1

T T-1 = I

1
0
T 1 =
0

0 0 dx
1 0 dy
0 1 dz

0 0
1

Scaling
Point in 3D given by (X1 Y1 Z1)
Scaled by (Sx Sy Sz)
X2 = X1 * Sx
Y2 = Y1 * Sy
Z2 = Z1 * Sz

Scaling
In matrix form
X 2 Sx 0 0 0 X 1
Y 0 Sy 0 0 Y
2=
1
Z 2 0 0 Sz 0 Z1


1 0 0 0 1 1

Inverse Transformation
Sx 0 0 0
0 Sy 0 0

S=
0 0 Sz 0

0
0
0
1

S S-1 = I

1
Sx

0
1
S =

0
0

1
Sy

0
0

1
Sz
0

0
1

Shearing
What will these do?

1
0

e 0 0
1 0 0

0 1 0

0 0 1

1
0

0 e 0
1 0 0

0 1 0

0 0 1

1
0

e 0 1
1 0 0 e

0 1 0 0

0 0 1 0

0 0
1 0 0

0 1 0

0 0 1

3D Rotation

Rotation about Z-axis

Z-coordinate will not change
Z = Z
If we ignore the Z-coordinate, it is 2-D
transformation in XY plane

(x2 y2 z2)

X ' cos
Y ' sin
=
Z ' 0

1 0

sin
cos
0
0

0 0 X
0 0 Y
1 0 Z

0 1 1

(x1 y1 z1)

Rotation
Y

Positive rotation is
counterclockwise when looking
down the axis of rotation
towards origin

(x2 y2 z2)

(x1 y1 z1)

Rotation about Principal Axes

Right hand rule
XYZ is a right handed system
if:
Place your right wrist at the
origin
curl your fingers from X to Y
thumb should point along +ve Z

Rotation about Principal Axes

Y

Rotation about Z

X
Z

x Rotation about Y

Rotation about X

y
x

z
y

X ' = X cos Y sin Y ' = Y cos Z sin Z ' = Z cos X sin

Y ' = X sin + Y cos Z ' = Y sin + Z cos X ' = Z sin + X cos
Z'= Z
X '= X
Y '= Y

Rotation about Principal Axes

X ' = X cos Y sin Y ' = Y cos Z sin Z ' = Z cos X sin
Y ' = X sin + Y cos Z ' = Y sin + Z cos X ' = Z sin + X cos
Z'= Z
X '= X
Y '= Y
Rotation about X

Rotation about Z
cos
sin

sin
cos
0
0

0
0
1
0

0
0
0

0
1
0 cos

0 sin

0
0

0
sin
cos
0

Rotation about Y
0
0
0

cos
0

sin

0 sin
1
0
0 cos
0
0

0
0
0

Concatenation of Rotations

Rotation around X by followed by rotation around Y

by followed by rotation around Z by

R = RZ RY RX

Properties of Rotation Matrix

Rotation Matrices are orthonormal and have a determinant of
+1
i.e. R R T = R TR = I

The inverse of a rotation matrix is its transpose

3D rigid motion can be described by a rotation and translation
X ' r11
Y ' = r
21
Z ' r31

r12
r22
r32

r13 X Tx
r23 Y + Ty
r33 Z Tz

or

X ' r11
Y ' r
= 21
Z ' r31

1 0

r12
r22
r32
0

r13 TX X
r23 Ty Y
r33 TZ Z

0 1 1

Properties of Rotation Matrix

r11
R = r21
r31

r12
r22
r32

r13
r23
r33

R has 9 unknowns, but orthonormality provides 6 constraints

0 i k
rij rkj = rji rjk =

1 i=k
j =1
j =1

Hence the number of degrees of freedom of a 3D rotation are 9

6=3
3

Properties of Rotation Matrices

Any concatenation of rotation matrices also forms a

rotation matrix i.e. the matrix remains orthonormal
[Proof?]
Quite simple,

Interpretation of Rotation Matrices

- 0.8256 0.40388 - 0.39404
- 0.20084 - 0.86294 - 0.46367

- 0.5273 - 0.30367 0.79356

How do I visualize this rotation?

39

Properties of Rotation Matrices

A rotation matrix transforms its own rows onto the
principal axes
r11 r12
r
r
21 22
r31 r32

r13 r11 1
r23 r12 = ?0

r33 r13 0

r11 r12
r
r
21 22
r31 r32

r13 r31 0
r23 r32 = 0

r33 r33 1

r11 r12
r
r
21 22
r31 r32

r13 r21 0
r23 r22 = 1

r33 r23 0

Rotation about Arbitrary Axis

Any complex rotation can be described by a single
rotation around an axis n by an angle
Therefore, any complex rotation can be described by
[n1, n2, n3]T and
Still 3 degrees of freedom as n can be taken to be a
unit vector without any loss of generality

n12 + n22 + n32 = 1

Rotation about Arbitrary Axis

To rotate about an axis n by an angle

1. Set up rotations such that n rotates onto one of the
principal axis [How?]
2. Rotate about that axis by
3. Undo the transformations in step 1

Rotation about Arbitrary Axis

Question: Given an arbitrary 3D rotation matrix, how
can we find out the axis n and the angle that
represents this rotation?
- 0.8256 0.40388 - 0.39404
- 0.20084 - 0.86294 - 0.46367

- 0.5273 - 0.30367 0.79356

Given R on the left, how

can we tell n and ?

Eigenvectors and Values of a Rotation Matrix

3D rotation matrix has eigenvalues of 1, cos + i sin
and cos -i sin
The eigenvector associated with the real eigenvalue
represents the axis of rotation [proof?]
We talked about this in the lecture conducted on 17 Nov.
2014
Think about what the Eigen Vectors definition is
And what effect Rotation has on the axis of rotation?

Summary: 3D Rotation Matrices

Any 3D rotation matrix can be described as

Rotations about Principal Axes

cos
sin

sin
cos
0
0

0 0
0 0
1 0

0 1

0
1
0 cos

0 sin

0
0

0
sin
cos
0

0
0
0

cos
0

sin

0 sin
1

0 cos
0

About origin, in right handed coordinate system,

counter clockwise when looking towards origin from
positive axis

Rotation matrix is orthonormal with

Determinant of +1 and 3 dof
Inverse of a rotation matrix is its transpose
Concatenation of Rotations is also a
rotation
IMP: A rotation matrix transforms its own
rows onto the principal axes

0
0
0

rotation about an axis n by an angle

To rotate about given axis n by :
Rotate axes onto a principal axis

Two ways: by computing principal

rotations or by composing appropriate
matrix through cross products
Rotate about principal axes and then undo the
earlier transformation

To compute n and from a 3D rotation

matrix
n is the eigenvector corresponding to the real
eigenvalue of 1
can be computed by the other 2 eigenvalues,
which are cos i sin

Hierarchy of 3D Transformations

Factorizing Transformations

Opposite of Concatenation of Transformations

Given a transformation matrix, decompose it into a
sequence of simpler transformations
Example:
a1
a
3
0

a2
a4
0

b1 1 0 b1 a1
b2 = 0 1 b2 a3
1 0 0 1 0

a2
a4
0

0
0
1

Question: How to factorize the multiplicative part?

a1
a
3

a2
a4

Is the factorization unique?

 Special Case: A is symmetric

Eigen values of a real symmetric matrix are real
Its eigenvectors can always be written as orthonormal matrix
A
=

A=
T
(Implication?)

A non symmetric real matrix M can be decomposed as

M = U S VT (with U and V being orthonormal, S being a diagonal
To compute U S and V,
Let A = MMT
A = (USVT)(USVT)T
A = US2UT
VT = (US)-1M

Singular Value Decomposition

Let M be a m-by-n matrix whose entries are real numbers. Then M
may be decomposed as
M = U S VT
where:
U is an m-by-m orthonormal matrix
S is an m-by-n matrix with non-negative numbers on the main diagonal and
zeros elsewhere
V is an n-by-n orthonormal matrix

Example
http://en.wikipedia.org/wiki/Singular_value_decomposition

Singular Value Decomposition

Implication: We can take the multiplicative part of any transform and
describe it as a sequence of a rotation, scaling and another rotation
2D Example: Decomposing an Affine Transformation
M = 0.95
0.23
0

0.49
0.89
0

>> U * S * V'

>> [U, S, V] = svd(M(1:2, 1:2))

U=
-0.78156
-0.62384

-0.62384
0.78156

S=

ans =
0.95
0.23

0.46
0.02
1

0.49
0.89

1.2904
0
0 0.56789
V=
-0.68658
-0.72705

-0.72705
0.68658

Interpretation in
terms of angles?

Singular Value Decomposition

Implications: Even a simple
shear can be written as a
rotationscalingrotation

Try it as an exercise

From Peter Shirley: Fundamentals of Computer Graphics

Summary: 2D and 3D Transformations

Image Registration
2D Transformations

Scaling
Shear
Rotation
Translation

Inverse Transformations
Rotation about an arbitrary point
Concatenation of transformations
Order of transformations
Factorization of Transformations

Displacement Models

Rigid / Euclidean
Similarity
Affine
Projective
Billinear, biquadratic etc

Recovering the best affine

transformation
Least Squared Error solution
Pseudo inverse

Image Warping
3D Transformations
Rotations about Principal Axes
Rotations about Arbitrary Axes

Properties of Rotation Matrices