Automatic Camera Calibration

Lu Zhang

Sep 22, 2005

Outline

Projective geometry and camera modals Principles of projective geometry Camera modals

Camera calibration methods Basic principles of self-calibration Stratified self-calibration

Projective Geometry Basic principles in projective spaces:

Points:A point of a dimensional projective space is represented by

an

n+1 vector of coordinates , are called

the homogeneous or projective coordinates of the points, x is called a

a coordinate vector. If for the two n+1 vectors represent the

same point.

np

121 ,...,, nxxxx ix

ii yx 11 ni

Projective Geometry The Projective Line The space is known as projective line. The standard projective basis of projective line is and . A point on the line

is , and are not both=0.

The point at infinity:

If let , when =0,

-> infinity, we call this point ‘point at infinity’

1p Te 0,11

Te 1,02 Te 1,13 2211 exexx 2x

2x21 eex 2

1

x

x

1x

Projective Geometry Projective space

Points:

Planes:

Lines:

A line is defined as the set of points that are linearly dependent on two points.

Plane at infinity: The points with =0 are said to be at infinity or ideal

points. The set of all ideal points may be written (X; Y; Z; 0). The set of all ideal points lies on a single plane, the plane at infinity.

Txxxxx 4321 ,,,

Tuuuuu 4321 ,,,

3P

4x

Camera models Camera models A point M on an object with coordinat

es (X,Y,Z) will be imaged at some point m=(x, y) in the image plane.

If consider the effect of focal length f, the relationship between image coordinate and 3-D space coordinate can be written as

here u=U/S v=V/S if S≠0, m=PMy

v

x

u

z

f

T

z

y

x

f

f

S

V

U

0100

000

000

Camera modals The general intrinsic matrix is

Intrinsic parameter indicates theproperty of camera itself. focal length pixel width pixel height, x coordinate at the optical

centre y coordinate at the optical

centre

0100

00

00

0

0

vfk

ufk

P v

u

vkuk

0u

0v

f

Camera models Camera Motion (Extrinsic parameters) If we go from the old coordinate system centered at C to

the new coordinate system centered at O by a rotation R followed by a translation T, in projective coordinates

The 4*4 matrix K is

newold KMM

103

T

tR

Camera models Intrinsic calibrationThe graph shows the transformation from retinal plane to itself.

The 3*3 matrix H is given by

Cause

We have

thus

oldnew Hmm

102

T

ts

MPm oldold

MHPm oldnew

KHPP oldnew

Self-calibration Self-calibration refers to the process of

calculating all the intrinsic parameters of the camera using only the information available in the images taken by that camera.

No calibration frame or known object is needed: the only requirement is that there is a static object in the scene, and the camera moves around taking images.

Self-calibration Why could we use self-calibratio

n? projective invariants

Epipolar geometry: The epipole is the point ofintersection of the linejoining the optical centerswith the image plane.

Thus the epipole is theimage, in one camera, of theoptical centre of the othercamera

Self-calibration a point x in one image gene

rates a line in the other on which its corresponding point x’ must lie.

With two views, the two camera coordinate systems are related by a rotation R and a translation T.

Self-calibration Therefore

Or if rewrite it as

Then we can define E, the essential matrix:

TRxx '

0)(' RxTx

0' Exx T

0

0

0

xy

xz

yz

tt

tt

tt

E

Self-calibration If we have two views of a point

M in three dimensional space, with M imaged at m in view 1 and m' in view 2

m=PM and m’=P’M If we set C(0,0,1), then we can

find p through PC=0,

where P=[P p] e’ is the projection of C on

view2, therefore

pPC 1

1''

1pPPe

Self-calibration

We also can get

Then can rewrite this equation as

mPPm 1''

1'

0''

'0'

''0

0'

PP

ee

ee

ee

F

Fmm

xy

xz

yz

T

Self-calibration A is the 3*3 left corner matrix of intrinsic matrix

P

We get the relationship between F and E

'

''

''

'

1

'

'

1 f

vv

uu

Av

u

and

f

vv

uu

Av

u

c

c

c

c

1' EAAF T

Self-calibration We can get three fundamenta

l matrices F By using the relationship bet

ween F,E and A, and the already known coordinate of epipolar points:

finally use a matrix equation

we can calculate the parameters in matrix A which is also the parameters in Matrix P, P is the intrinsic Matrix which include camera characteristics.

323123211312 ,,,,, eeeeee

Stratified self-calibration

What is Stratified self-calibration? Self-calibration is the process of determining internal cam

era parameters directly from multiple uncalibrated images.

First, Stratified self-calibration performs a projective reconstruction. Then, it obtains an affine reconstruction as the initial value. Finally, it applies metric reconstruction.

Stratification of geometry Projective Affine Metric

Stratified self-calibration Obtaining Projective camera matrix Determine the rectifying homography H is the projective camera matrix for each view i For the actual cameras, the internal parameter K is the same for each

view, but in general the calibration matrix will differ for each view. Therefore, the purpose for self-calibration is to find a rectifying homo

graphy H

Obtaining the metric reconstruction

iP

iP][ iiii tRKP

iK

][ iii tIKRHP

},{},{ 1j

ij

i XHHPXP

Stratified self-calibration Stratified self-calibration algorithm Step 1: affine calibration

a. formulate the modulus constraint for all pairs of views, need at least 3 views

b. (for n>3) solve the set of equations c. compute the affine projection matrices

Step 2: metric calibrationa. compute the position of plane at infinityb. find the intrinsic parameters Kc. compute the metric projection matrices

Pipeline Projective reconstruction

Relating images Initial reconstruction Adding views

Self-calibration Finding plane at infinity Compute K Metric reconstruction

Dense depth estimation Rectification Dense stereo matching

Modeling

Projective reconstruction Relating images

Detecting feature points Harris corner detector

Matching feature points RANSAC (RANdom Sampling Consensus) Determine Fundamental MatrixLeast square estimation

The whole procedure: Repeat

a. take minimal sample(8)b. compute Fc. estimate inliers

Until (inliers, trials)> 95% Refine F (using all inliers)

OKP

Projective reconstruction Results from projective reconstruction

Feature points detection (the original images are captured from a video sequence)

After applying Harris Corner Detector

Projective reconstruction Improvement by using RANSAC

Computation of Fundamental Matrix by applying least square estimation

0.2005- 0.0218- 0.0132

0.0218 0.0000 0.0000-

0.0121- 0.0000 0.0000-

F

Projective reconstruction Result for projective reconstruction (con’t)

Feature points detection

After applying Harris Corner Detector

Figure 9 Figure 10

Projective reconstruction Improvement by using RANSAC

Computation of Fundamental Matrix by applying least square estimation

0.1853 0.0043 0.0006

0.0044- 0.0000 0.0000-

0.0007- 0.0000 0.0000-

F

Self-calibration Finding initial Projective Matrix

Compute epipoles and epipolar lines Epipolar lines: & Epipoles: on all epipolar lines, thus x ,

Fxl '

]λe'|ve'F][[e'P' 0]|[IP T

'xFl T

0' Fxe T

0' Fe T 0Fe

Self-calibration Result from Initial Projective Matrix

Epipolar lines

Epipoles

0.0017

0.4982

0.8670

1e

0.0018

0.5067

0.8621

2e

Self-calibration Result from Initial Projective Matrix

Epipolar lines

Epipoles

0.0004-

0.1685-

0.9857

1e

0.0004-

0.1182-

0.9930

2e

Self-calibration Projective Matrix

0.0018 0.0250 0.0000- 0.0000-

0.5067 0.1729 0.0188 0.0114-

0.8621 0.1016- 0.0110- 0.0067

2P

0.0004- 0.0044- 0.0000 0.0000-

0.1182- 0.1840- 0.0042- 0.0006-

0.9930 0.0219- 0.0005- 0.0001-

2P

1.0000 0 0

0.1590 695.6221 0

0.5541 59.7937 677.2568

K

0.1853 0.0043 0.0006

0.0044- 0.0000 0.0000-

0.0007- 0.0000 0.0000-

F

Thanks guys, any questions?