Camera Calibration

(Trucco, Chapter 6)

What is the goal of camera calibration?

- To produce an estimate of the extrinsic and intrinsic camera parameters.

Procedure
- Given the correspondences between a set of point features in the world (X w , Y w , Z w ) and their projections in an image (x im , y im ), compute the intrinsic and extrinsic camera parameters.
x im pixel frame yim P

Yc Camera Frame Xc
optical axis

y Yw o ,oy x image plane frame x World Frame

principal point

Xw Zw

Zc
center of perspective projection

Establishing the correspondences

- Calibration methods rely on one or more images of a calibration pattern: (1) a 3D object of known geometry. (2) it is located in a known position in space. (3) it is generating image features which can be located accurately.

-2-

- Consider the above calibration pattern: * it consists of two orthogonal grids. * equally spaced black squares drawn on white, perpendicular planes. * assume that the world reference frame is centered at the lower left corner of the left grid, with axes parallel to the three directions identied by the calibration pattern. * given the size of the planes, their angle, the number of squares etc. (all known by construction), the coordinates of each vertex can be computed in the world reference frame using trigonometry. * the projection of the vertices on the image can be found by intersecting the edge lines of the corresponding square sides (or through corner detection).

-3-

Methods
(1) Direct parameter calibration. Direct recovery of the intrinsic and extrinsic camera parameters. (2) Camera parameters through the projection matrix M = M in M ex m 11 = m 21 m 31 m 12 m 22 m 32 m 13 m 23 m 33 m 14 m 24 m 34

(2.1) Estimate the elements of the projection matrix. (2.2) Compute the intrinsic/extrinsic as closed-form functions of the entries of the projection matrix.

-4-

Method 1: Direct Parameter Calibration

- We assume that the world reference frame is known (e.g., the origin is the middle lower corner of the calibration pattern).

Review of basic equations

- From world coordinates to camera coordinates (note that we have changed the order of rotation/translation): P c = R(P w T ) or P c = RP w RT or P c = RP w T - In the rest of this discussion, I will replace T with T : X c r 11 Y = r c 21 Z c r 31 r 12 r 22 r 32 r 13 X w T x r 23 Y w + T y r 33 Z w T z

- From camera coordinates to pixel coordinates: x im = x/s x + o x = y im = y/s y + o y = f Xc + ox s x Zc f Yc + oy s y Zc

- Relating world coordinates to pixel coordinates: x im o x = f /s x y im o y = f /s y r 11 X w + r 12Y w + r 13 Z w + T x r 31 X w + r 32Y w + r 33 Z w + T z r 21 X w + r 22Y w + r 23 Z w + T y r 31 X w + r 32Y w + r 33 Z w + T z

-5-

Independent intrinsic parameters

- The ve intrinsic parameters f , s x , s y , o x , o y are not independent. - We can dene the following four independent parameters: f x = f /s x , the focal length in horizontal pixels = s y /s x (or = f x / f y ), aspect ratio (o x , o y ), image center coordinates

Method 1: main steps

(1) Assuming that o x and o y are known, estimate all the remaining parameters. (2) Estimate o x and o y

Step 1: estimate f x , , R, and T

- To simplify notation, consider (x im o x , y im o y ) = (x, y) x = fx y = fy r 11 X w + r 12Y w + r 13 Z w + T x r 31 X w + r 32Y w + r 33 Z w + T z r 21 X w + r 22Y w + r 23 Z w + T y r 31 X w + r 32Y w + r 33 Z w + T z

- Using the fact that the above two equations have the same denominator, we get the following equation: x f y (r 21 X w + r 22Y w + r 23 Z w + T y ) = y f x (r 11 X w + r 12Y w + r 13 Z w + T x ) Problem Statement Assuming that o x and o y are known, compute f x , , R, and T from N corresponding pairs of points (X iw , Y iw , Z iw ), (x i , y i ), i = 1, . . . , N .

-6Derive a system of equations - Each pair of corresponding points leads to an equation: x i f y (r 21 X iw + r 22Y iw + r 23 Z iw + T y ) = y i f x (r 11 X iw + r 12Y iw + r 13 Z iw + T x ) - Rewrite the above equation as follows (i.e., divide by f y ): x i X iw v 1 + x i Y iw v 2 + x i Z iw v 3 + x i v 4 y i X iw v 5 y i Y iw v 6 y i Z iw v 7 y i v 8 = 0 where v1 v2 v3 v4 = r 21 = r 22 = r 23 = Ty v5 = v6 = v7 = v8 =

r 11 r 12 r 13 Tx

- N corresponding points lead to a homogeneous system of N equations with 8 unknowns: Av = 0 where:

w x1 X1 w x2 X2 A= ... w xN XN w x 1Y 1 w x 2Y 2 ... w xNYN w x1 Z1 w x2 Z2 ... w xN ZN

x1 x2 ... xN

w y 1 X 1 w y 2 X 2 ... w y N X N

w y 1Y 1 w y 2Y 2 ... w y N Y N

w y 1 Z 1 w y 2 Z 2 ... w y N Z N

y 1 y 2 ... y N

Solving the system - It can be shown that if N 7, then A has rank 7. - If A = UDV T , we have discussed in class that the system has a nontrivial solution v which is proportional to the column of V corresponding to the smallest singular value of A (i.e., the last column of V which we denote as v): v = v ( is the scale factor) or v = v ( = 1/ ) - Using the components of v and v: (v 1 , v 2 , v 3 , v 4 , v 5 , v 6 , v 7 , v 8 ) = (r 21 , r 22 , r 23 , T y , r 11 , r 12 , r 13 , T x )

-7

Determine

and | |

3 2 2 2 v2 + v2 + v2 = 2 (r 21 + r 22 + r 23 ) = | 1 2
2 2 2 (r 21 + r 22 + r 23 = 1) 2 2 7 2 v2 + v2 + v2 = 2 2 (r 11 + r 12 + r 13 ) = 6 5 2 2 2 (r 11 + r 12 + r 13 = 1 and >0)

| |

Determine r 21 , r 22 , r 23 , r 11 , r 12 , r 13 , T y , T x - We can determine the above parameters, up to an unknown common sign.

Determine r 31 , r 32 , r 33 - Can be estimated as the cross product of R1 and R2 : R3 = R1 x R2 - The sign of R3 is already xed (the entries of R3 remain unchanged if the signs of all the entries of R1 and R2 are reversed). Ensuring the orthogonality of R - The computation of R does not take into account explicitly the orthogonality constraints. T - The estimate R of R cannot be expected to be orthogonal (e.g., R R = I ). - We can "enforce" the orthogonality on R by using its SVD: R = UDV T - Replace D with I , e.g., R = UIV T ( R RT = I )

r 21 = 1/| r 22 = 1/| r 23 = 1/| T y = 1/|

| v1 | v2 | v3 | v4

r 11 = 1/ r 12 = 1/ r 13 = 1/ T x = 1/

| | | |

| v5 | v6 | v7 | v8

-8

Determine the sign of

- Consider the following equations again: x = f /s x y = f /s y r 11 X w + r 12Y w + r 13 Z w + T x Xc = f /s x r 31 X w + r 32Y w + r 33 Z w + T z Zc r 21 X w + r 22Y w + r 23 Z w + T y Yc = f /s y Zc r 31 X w + r 32 r w + r 33 Z w + T z

- If Z c >0, then x and r 11 X w + r 12Y w + r 13 Z w + T x must have opposite signs (it is sufcient to check the sign for one of the points). if x(r 11 X w + r 12Y w + r 13 Z w + T x ) > 0, then reverse the signs of r 11 , r 12 , r 13 , and T x else no further action is required - Similarly, if Z c >0, then y and r 21 X w + r 22Y w + r 23 Z w + T x must have opposite signs (it is sufcient to check the sign for one of the points). if y(r 21 X w + r 22Y w + r 23 Z w + T y ) > 0, then reverse the signs of r 21 , r 22 , r 23 , and T y else no further action is required

-9Determine T z and f x : - Consider the equation: x = f /s x - Lets rewrite it in the form: x(r 31 X w + r 32Y w + r 33 Z w + T z ) = f /s x (r 11 X w + r 12Y w + r 13 Z w + T x ) - We can obtain T z and f x by solving a system of equations like the above, written for N points: Tz A = b where fx x1 x2 A= ... xN
w w w w w w (r 11 X 1 + r 12Y 1 + r 13 Z 1 + T x ) x 1 (r 31 X 1 + r 32Y 1 + r 33 Z 1 + T x ) w w w w w w x 2 (r 31 X 2 + r 32Y 2 + r 33 Z 2 + T x ) (r 11 X 2 + r 12Y 2 + r 13 Z 2 + T x ) b= ... ... w w w w w w (r 11 X N + r 12Y N + r 13 Z N + T x ) x N (r 31 X N + r 32Y N + r 33 Z N + T x )

r 11 X w + r 12Y w + r 13 Z w + T x r 31 X w + r 32Y w + r 33 Z w + T z

- Using SVD, the (least-squares) solution is: Tz T 1 T = ( A A) A b fx Determine f y : - From f x = f /s x and f y = f /s y we have:

fy = fx/

-10-

Step 2: estimate o x and o y

- The computation of o x and o y will be based on the following theorem: Orthocenter Theorem: Let T be the triangle on the image plane dened by the three vanishing points of three mutually orthogonal sets of parallel lines in space. The image center (o x , o y ) is the orthocenter of T . - We can use the same calibration pattern to compute three vanishing points (use three pairs of parallel lines dened by the sides of the planes).

Note 1: it is important that the calibration pattern is imaged from a viewpoint guaranteeing that none of the three mutually orthogonal directions will be near parallel to the image plane ! Note 2: to improve the accuracy of the image center computation, it is a good idea to estimate the center using several views of the calibration pattern and average the results.

-11-

Method 2: Camera parameters through the projection matrix

Review of basic equations
xh y = M M in ex h w x= y= Xw m Xw Y w 11 Yw = m 21 =M Zw Zw m 31 1 1 m 12 m 22 m 32 m 13 m 23 m 33 Xw m 14 Yw m 24 Zw m 34 1

x h m 11 X w + m 12Y w + m 13 Z w + m 14 = m 31 X w + m 32Y w + m 33 Z w + m 34 w y h m 21 X w + m 22Y w + m 23 Z w + m 24 = m 31 X w + m 32Y w + m 33 Z w + m 34 w

(Note: I have replaced x im with x and y im with y for simplicity)

Step 1: solve for mij s

- The matrix M has 11 independent entries (e.g., divide every entry by m 11 ). - We would need at least N =6 world-image point correspondences to solve for the entries of M. m 11 X iw + m 12Y iw + m 13 Z iw + m 14 m 31 x i X iw m 32 x i Y iw m 33 x i Z iw + m 34 = 0 m 21 X iw + m 22Y iw + m 23 Z iw + m 24 m 31 y i X iw m 32 y i Y iw m 33 y i Z iw + m 34 = 0 - These equations will lead to a homogeneous system of equations: Am = 0 where X1 0 Xw 2 A= 0 ... w XN 0
w w Y1 0 w Y2 0 ... w YN 0 w Z1 0 w Z2 0 ... w ZN 0

1 0 1 0 ... 1 0

0 w X1 0 w X2 ... 0 w XN

0 w Y1 0 w Y2 ... 0 w YN

0 w Z1 0 w Z2 ... 0 w ZN

w 0 x 1 X 1 w 1 y 1 X 1 w 0 x 2 X 2 w 1 y 2 X 2 ... ... w 0 x N X N w 1 y N X N

w x 1Y 1 w y 1Y 1 w x 2Y 2 w y 2Y 2 ... w x N Y N w y N Y N

w x 1 Z 1 w y 1 Z 1 w x 2 Z 2 w y 2 Z 2 ... w x N Z N w y N Z N

x 1 y 1 x 2 y 2 ... x N y N

-12- It can be shown that A has rank 11 (for N 11). - If A = UDV T , the system has a nontrivial solution m which is proportional to the column of V corresponding to the smallest singular value of A (i.e., the last column of V denoted here as m): m = m ( is the scale factor) or m = m ( = 1/ )

Step 2: nd the intrinsic/extrinsic parameters using mij s

- The full expression for M is as follows: f x r 11 + o x r 31 M = f y r 21 + o y r 31 r 31 f x r 12 + o x r 32 f y r 22 + o y r 32 r 32 f x r 13 + o x r 33 f y r 23 + o y r 33 r 33 f xT x + oxTz f yT y + oyTz Tz

- Lets dene the following vectors: q 1 = (m 11 , m 12 , m 13 )T q 2 = (m 21 , m 22 , m 23 )T q 3 = (m 31 , m 32 , m 33 )T q 4 = (m 14 , m 24 , m 34 )T - The solutions are as follows (see book for details): o x = qT q3 1 f x = q T q 1 o2 x 1 - The rest parameters are easily computed .... Question: how would you estimate the accuracy of a calibration algorithm? o y = qT q3 2 f y = q T q 2 o2 2 y

-13-

Some comments
- The precision of calibration depends on how accurately the world and image points are located. - Studying how localization errors "propagate" to the estimates of the camera parameters is very important. - Although the two methods described here should produce the same results (at least theoretically), we usually obtain different solutions due to different error propagations. - Method 2 is simpler and should be preferred if we do not need to compute the intrinsic/extrinsic camera parameters explicitly.