(geometric) camera calibration - purdue university(geometric) camera calibration cs635 spring 2010...

(Geometric) Camera Calibration(Geometric) Camera Calibration

CS635 Spring 2010

Daniel G AliagaDaniel G. AliagaDepartment of Computer Science

Purdue University

Camera CalibrationCamera CalibrationCamera CalibrationCamera Calibration

• Digital Cameras and CCDsDigital Cameras and CCDs

• Aberrations• Aberrations

P ti P j ti• Perspective Projection

• Calibration

Digital Camera vs. “Film” CameraDigital Camera vs. “Film” Camera

• Charge‐Coupled Device (CCD)

Digital Camera vs.  Film  CameraDigital Camera vs.  Film  Camera

– Image plane is a CCD array instead of film– CCD arrays are typically ¼ or ½ inch in size– CCD arrays have a pixel resolution (e.g., 640x480, 1024x1024)y p ( g , , )– CCD Cameras have a maximum “frame rate”, usually determined by 

the hardware and bandwidth

• Number of CCDs– 3: each CCD captures only R, G, or B wavelengths– 1: the single CCD captures RGB simultaneously, reducing the 

resolution by 1/3 (kinda)y / ( )

• Video– Interlaced: only “half” of the horizontal lines of pixels are present in 

each frameeach frame– Progressive scan: each frame has a full‐set of pixels

The simplest 1The simplest 1‐‐CCD CCD camera in towncamera in town

image plane

objectCCD array

object

ExposuresExposuresExposuresExposures

apertureimage plane

objectCCD array

object

“effective” shutter ( ft j t t(e.g., often just turn

CCD on/off)

ExposuresExposuresExposuresExposures

• An “exposure” is when the CCD is exposed to theAn  exposure  is when the CCD is exposed to the scene, typically for a brief amount of time and with a particular set of camera parameters

• The characteristics of an “exposure” are determined pby multiple factors, in particular:– Camera aperture

• Determines amount of light that shines onto CCD

– Camera shutter speed• Determines time during which aperture is “open” and light shinesDetermines time during which aperture is  open  and light shines on CCD

Camera CalibrationCamera CalibrationCamera CalibrationCamera Calibration

• Digital Cameras and CCDsDigital Cameras and CCDs

• Aberrations• Aberrations

P ti P j ti• Perspective Projection

• Calibration

AberrationsAberrationsAberrationsAberrations

• A “real” lens system does not produce a perfectA  real  lens system does not produce a perfect image

• Aberrations are caused by imperfect manufacturing and by our approximate modelsy pp– Lenses typically have a spherical surface

• Aspherical lenses would better compensate for refraction but are more difficult to manufacturemore difficult to manufacture

– Typically 1st order approximations are used• Remember sin Ω = Ω ‐Ω3/3! + Ω5/5! ‐ …

• Thus, thin‐lens equations only valid iff sin Ω ≈ Ω

AberrationsAberrationsAberrationsAberrations

• Most common aberrations:Most common aberrations:– Spherical aberration

Coma– Coma

– Astigmatism

C t f fi ld– Curvature of field

– Chromatic aberration

Di i– Distortion

Spherical AberrationSpherical Aberration

D t i t i l i

Spherical AberrationSpherical Aberration

• Deteriorates axial image

ComaComaComaComa

• Deteriorates off‐axial bundles of raysDeteriorates off axial bundles of rays

Astigmatism and Curvature of FieldAstigmatism and Curvature of Field

• Produces multiple (two) images of a single

Astigmatism and Curvature of FieldAstigmatism and Curvature of Field

Produces multiple (two) images of a single object point

Chromatic AberrationChromatic AberrationChromatic AberrationChromatic Aberration

• Caused by wavelength dependent refractionCaused by wavelength dependent refraction– Apochromatic lenses (e.g., RGB) can help

DistortionDistortion

• Radial (and tangential) image distortions

Example radial distortions

Radial DistortionRadial DistortionRadial DistortionRadial Distortion

• (x, y) pixel before distortion correction( , y) p

• (x’, y’) pixel after distortion correction

• Let r = (x2 + y2) ‐1

• Then– x’ = x(1 ‐ Δr/r)x  x(1  Δr/r)– y’ = y(1 ‐ Δr/r)– where Δr = k0r + k1r3 + k2r5 + …

• Finally,– x’ = x(1 – k0 – k1r2 – k2r4 ‐ …)x  x(1  k0 k1r k2r …)

– y’ = y(1 – k0 – k1r2 – k2r4 ‐ …)

Radial DistortionRadial DistortionRadial DistortionRadial Distortion

before after

Camera CalibrationCamera CalibrationCamera CalibrationCamera Calibration

• Digital Cameras and CCDsDigital Cameras and CCDs

• Aberrations• Aberrations

P ti P j ti• Perspective Projection

• Calibration

Perspective ProjectionPerspective ProjectionPerspective ProjectionPerspective Projection

zZ

(X, Y, Z)fzZ

/ i i t

(x, y)

optical axis

Yy?

image planeeye/viewpoint optical axis

f Zx = f X

Zy = f Y

Zy = Y &

Perspective ProjectionPerspective ProjectionPerspective ProjectionPerspective Projection

Z

image plane

f

Z

(X, Y, Z)plane

bj t(x, y) object(x, y)

x = f X y = f Yx = f XZ

y = f YZ

Camera CalibrationCamera CalibrationCamera CalibrationCamera Calibration

• Digital Cameras and CCDsDigital Cameras and CCDs

• Aberrations• Aberrations

P ti P j ti• Perspective Projection

• Calibration

Tsai Camera Model and CalibrationTsai Camera Model and CalibrationTsai Camera Model and CalibrationTsai Camera Model and Calibration

• A widely used camera model to calibrateA widely used camera model to calibrate conventional cameras based on a pinhole cameracamera

• Reference• Reference– “A Versatile Camera Calibration Technique for High Accuracy 3D Machine Vision Metrology UsingHigh‐Accuracy 3D Machine Vision Metrology Using Off‐the‐Shelf TV Cameras and Lenses”, Roger Y. Tsai, IEEE Journal of Robotics and Automation,Tsai,  IEEE Journal of Robotics and Automation, Vol. 3, No. 4, August 1987

Calibration GoalCalibration GoalCalibration GoalCalibration Goal

• Determine the intrinsic (and extrinsic)Determine the intrinsic (and extrinsic) parameters of a camera (with lens)

22

Camera ParametersCamera ParametersCamera ParametersCamera Parameters

• Intrinsic/InternalIntrinsic/Internal– Focal length

l ( )

f– Principal point (center)

– Pixel size yx pp ,

yx ss ,

– (Distortion coefficients)

• Extrinsic/External

,...1k

Extrinsic/External– Rotation

Translation

ψϕφ ,,ttt

23

– Translation zyx ttt ,,

Focal LengthFocal LengthFocal LengthFocal Length

⎟⎞

⎜⎛⎤⎡

⎟⎞

⎜⎛ 000

XffX

⎞⎛⎞⎛

⎟⎟⎟⎟⎟

⎠⎜⎜⎜⎜⎜

⎝⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

10100000000

ZY

ff

ZfYfX

⎟⎟⎠

⎞⎜⎜⎝

⎛ZfYZfX

//

=⎟⎟⎠

⎞⎜⎜⎝

⎛yx

⎟⎠

⎜⎝⎦⎣⎠⎝ 1

Principal PointPrincipal PointPrincipal PointPrincipal Point

⎟⎟⎞

⎜⎜⎛

⎥⎤

⎢⎡

⎟⎞

⎜⎛ + 00

YX

pfZpfX xx⎞⎛

⎟⎟⎟⎟

⎠⎜⎜⎜⎜

⎝⎥⎥⎥

⎦⎢⎢⎢

⎣

=⎟⎟⎟

⎠⎜⎜⎜

⎝

+

1010000

ZY

pfZ

ZpfY yy⎟⎟⎠

⎞⎜⎜⎝

⎛

0

0

yx

⎠⎝

CCD Camera: Pixel SizeCCD Camera: Pixel SizeCCD Camera: Pixel SizeCCD Camera: Pixel Size

⎤⎡⎤⎡ 0000 pfs

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

01000000

1000000

y

x

y

x

pfpf

ss

K

⎦⎣⎦⎣

⎥⎥⎤

⎢⎢⎡

= 0000 xx

pp

K αα

(intrinsic) calibration matrix⎥⎥⎦⎢

⎢⎣ 0100

00 yy pK α (intrinsic) calibration matrix

Translation & RotationTranslation & RotationTranslation & RotationTranslation & Rotation

⎟⎟⎞

⎜⎜⎛⎤⎡ tR Y

X)~(~ CXRxcam −= ψϕφ RRRR=

⎟⎟⎟⎟

⎠⎜⎜⎜⎜

⎝

⎥⎦

⎤⎢⎣

⎡=

110tR~

ZY

xcamRCXRxcam −=~~

t−

3x3 rotation matrices

[ ]Tzyx tttt=⎠⎝ 1 [ ]zyxtranslation vector

(extrinsic) calibration matrix P

Calibration TaskCalibration TaskCalibration TaskCalibration Task

points in space

observation

physical arrangement (calibration pad)

(camera with initial parameters)

calibration result(camera with calibrated

? ?

( p ) (parameters)

~~ KG PWhat is ? ?ii x~X ↔ KGiven P

A Linear FormulationA Linear FormulationA Linear FormulationA Linear Formulation

KPM =Let KPM =

XMxcam~~ =

⎟⎞

⎜⎛⎤⎡⎞⎛ '

X

Let

⎟⎟⎟⎟⎟⎞

⎜⎜⎜⎜⎜⎛

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

'''

34333231

24232221

14131211

ZY

mmmmmmmmmmmm

wyx

⎟⎟⎠

⎞⎜⎜⎝

⎛=⎟⎟

⎠

⎞⎜⎜⎝

⎛'/''/'

wywx

yx

⎟⎟⎠

⎜⎜⎝⎥⎦⎢⎣⎠⎝ 134333231 mmmmw

)~/()~( 31 XmXmx ⋅⋅= )()( 31

)~/()~( 32 XmXmy ⋅⋅=

A Linear FormulationA Linear FormulationA Linear FormulationA Linear Formulation

for observationsni ...1=)~/()~( 31 XmXmx ⋅⋅=o obse a o s

0~)( X

)~/()~( 32 XmXmy ⋅⋅=

0)( 31 =⋅− ii Xmxm 2n homogeneous linear equations and 12 unknowns (coefficients of M)0~)( 32 =⋅− ii Xmym

Thus, given n≥6 can solve for M; namely 0=Qm

⎥⎤

⎢⎡ − TTT XxX ~0~

111

⎥⎥⎥⎥⎥

⎢⎢⎢⎢⎢

−=

TTT

TTT XyXQ

~~......

~~0 111

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛= 2

1

mm

m

⎥⎥

⎦⎢⎢

⎣ −−

Tnn

TT

Tnn

TTn

XynXXxX~~0

0⎟⎠

⎜⎝ 3m

Decomposing M into Camera Decomposing M into Camera ParametersParameters

[ ] [ ]tRKbAM == ρ

⎥⎥⎤

⎢⎢⎡

++−

=⎥⎥⎤

⎢⎢⎡

= Ty

Ty

Tx

TTx

T

T

rprrprr

aa

A 32

321

2

1

)sin/()cot(

θαθαα

ρ⎥⎥

⎦⎢⎢

⎣⎥⎥

⎦⎢⎢

⎣T

yyT r

pa 3

32

3

2 )(ρ

…function of yxyx pp ,,,αα and skew θyy

(often we assume skew is non-existent, which means )2/πθ =

Decomposing M into Camera Decomposing M into Camera ParametersParameters

2

233

3

)(

/ar

a=

=

ρ

ερ

aa

aa

y

x

322

312

sin

sin

×=

×=

θρα

θρα

322

312

)()(

)(

)(

aap

aap

y

x

⋅=

⋅=

ρ

ρ

aaaar

32

321

×

××

=

3231

3231 )()(cosaaaaaaaa

×××⋅×

−=θbKt

rrr1

132−=

×=

ρ

⎥⎤

⎢⎡

=⎥⎥⎤

⎢⎢⎡

= ,0000

tRPp

pK yy

xx

αα

⎥⎦

⎢⎣⎥

⎥⎦⎢

⎢⎣

10,

0100pyy

A Nonlinear FormulationA Nonlinear FormulationA Nonlinear FormulationA Nonlinear Formulation

• Bundle AdjustmentBundle Adjustment– Given initial guesses, use nonlinear least squares to refine/compute the calibration parametersto refine/compute the calibration parameters

– Simple but good convergence depends on accuracy of initial guessaccuracy of initial guess

A Nonlinear FormulationA Nonlinear FormulationA Nonlinear FormulationA Nonlinear Formulation

)~/()~( XXRecall

)/()( 31 XmXmx ⋅⋅=

)~/()~( 32 XmXmy ⋅⋅=

∑⎥⎥⎦

⎤

⎢⎢⎣

⎡

⋅

⋅−+

⋅

⋅−=

ij ji

jiij

ji

jiij Xm

Xmy

XmXm

xmn

E 2

3

22

3

1 )~~

()~~

(1

Goal is 0→E

A Nonlinear FormulationA Nonlinear FormulationA Nonlinear FormulationA Nonlinear FormulationOption A:

Define as a matrix of 11 unknowns (i.e., ) M 134 =mAnd solve for ijm

Can be made very efficient, especially for sparse matrices

Option B:

Define as function of intrinsic and extrinsic parameters so that it is “recomputed” during each loop

Mparameters so that it is recomputed during each loop of the optimization