camera calibration
DESCRIPTION
Camera calibration. Digital Visual Effects, Spring 2007 Yung-Yu Chuang 2007/4/17. with slides by Richard Szeliski, Steve Seitz, and Marc Pollefyes. Announcements. Project #2 is due next Tuesday before the class. Outline. Camera projection models Camera calibration (tools) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/1.jpg)
Camera calibration
Digital Visual Effects, Spring 2007Yung-Yu Chuang2007/4/17
with slides by Richard Szeliski, Steve Seitz, and Marc Pollefyes
![Page 2: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/2.jpg)
Announcements• Project #2 is due next Tuesday before the
class
![Page 3: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/3.jpg)
Outline• Camera projection models• Camera calibration (tools)• Nonlinear least square methods• Bundle adjustment
![Page 4: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/4.jpg)
Camera projection models
![Page 5: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/5.jpg)
Pinhole camera
![Page 6: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/6.jpg)
Pinhole camera model
(optical center)
origin
principal point
P(X,Y,Z)
p
(x,y)
![Page 7: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/7.jpg)
Pinhole camera model
10100000000
~1
ZYX
ff
ZfYfX
yx
ZfYy
ZfXx
principal point
![Page 8: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/8.jpg)
Pinhole camera model
1010000100001
1000000
~1
ZYX
ff
ZfYfX
yx
principal point
![Page 9: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/9.jpg)
Principal point offset
1010000100001
1000
0~
10
0
ZYX
yfxf
ZfYfX
yx
XIKx 0~
intrinsic matrix
principal point
only related to camera projection
![Page 10: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/10.jpg)
Intrinsic matrix
• non-square pixels (digital video)• skew• radial distortion
1000
0
0
0
yfxf
K
1000 0
0
yfxsfa
K
Is this form of K good enough?
![Page 11: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/11.jpg)
Distortion
• Radial distortion of the image– Caused by imperfect lenses– Deviations are most noticeable for rays that
pass through the edge of the lens
No distortion Pin cushion Barrel
![Page 12: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/12.jpg)
Camera rotation and translation
tR
ZYX
ZYX
33
'''
1100
00
~1
0
0
ZYX
yfxf
yx
tR
XtRKx ~
extrinsic matrix
![Page 13: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/13.jpg)
Two kinds of parameters• internal or intrinsic parameters such as
focal length, optical center, aspect ratio:what kind of camera?
• external or extrinsic (pose) parameters including rotation and translation:where is the camera?
![Page 14: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/14.jpg)
Other projection models
![Page 15: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/15.jpg)
Orthographic projection• Special case of perspective projection
– Distance from the COP to the PP is infinite
– Also called “parallel projection”: (x, y, z) → (x, y)
Image World
![Page 16: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/16.jpg)
Other types of projections• Scaled orthographic
– Also called “weak perspective”
• Affine projection– Also called “paraperspective”
![Page 17: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/17.jpg)
Fun with perspective
![Page 18: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/18.jpg)
Perspective cues
![Page 19: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/19.jpg)
Perspective cues
![Page 20: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/20.jpg)
Fun with perspective
Ames room
![Page 21: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/21.jpg)
Forced perspective in LOTR
![Page 22: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/22.jpg)
Camera calibration
![Page 23: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/23.jpg)
Camera calibration• Estimate both intrinsic and extrinsic
parameters• Mainly, two categories:1. Photometric calibration: uses reference
objects with known geometry2. Self calibration: only assumes static scene,
e.g. structure from motion
![Page 24: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/24.jpg)
Camera calibration approaches1. linear regression (least squares)2. nonlinear optimization3. multiple planar patterns
![Page 25: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/25.jpg)
Chromaglyphs (HP research)
![Page 26: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/26.jpg)
Linear regression
MXXtRKx ~
![Page 27: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/27.jpg)
Linear regression• Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)
![Page 28: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/28.jpg)
Linear regression
![Page 29: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/29.jpg)
Linear regression
![Page 30: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/30.jpg)
Linear regression
Solve for Projection Matrix M using least-square techniques
![Page 31: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/31.jpg)
Normal equationGiven an overdetermined system
bAx
bAAxA TT
the normal equation is that which minimizes the sum of the square differences between left and right sides
Why?
![Page 32: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/32.jpg)
Normal equation
2)( bAxx E
n
m
nmn
m
b
b
x
x
aa
aa
:::
:
...::::::
... 1
1
1
111
nxm, n equations, m variables
![Page 33: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/33.jpg)
Normal equation
n
m
jjnj
i
m
jjij
m
jjj
n
i
m
jjnj
m
jjij
m
jjj
bxa
bxa
bxa
b
b
b
xa
xa
xa
1
1
11
11
1
1
11
:
:
:
:
:
:
bAx
n
ii
m
jjij bxaE
1
2
1
2)( bAxx
![Page 34: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/34.jpg)
Normal equation
n
ii
m
jjij bxaE
1
2
1
2)( bAxx
1
0xE
n
iii
n
ij
m
jiji baxaa
11
1 11 22
11 1
2 i
n
ii
m
jjij abxa
)(20 bAAxAx
TT
E bAAxA TT →
![Page 35: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/35.jpg)
Normal equation
bbxbAxbAAxAx
bbbAxAxbAxAxbAxbAxbAxbAx
bAxbAx
bAx
TTTTTTT
TTTTTT
TTT
TT
T
2
bAAxAx
TT 22 E
![Page 36: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/36.jpg)
Linear regression• Advantages:
– All specifics of the camera summarized in one matrix
– Can predict where any world point will map to in the image
• Disadvantages:– Doesn’t tell us about particular parameters– Mixes up internal and external parameters
• pose specific: move the camera and everything breaks
![Page 37: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/37.jpg)
Nonlinear optimization• A probabilistic view of least square• Feature measurement equations
• Likelihood of M given {(ui,vi)}
![Page 38: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/38.jpg)
Optimal estimation• Log likelihood of M given {(ui,vi)}
• It is a least square problem (but not necessarily linear least square)• How do we minimize C?
![Page 39: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/39.jpg)
Optimal estimation• Non-linear regression (least squares), because the relations between ûi and ui are non-linear functions M
• We can use Levenberg-Marquardt method to minimize it
XtRKuuu ~ˆ
known constant
We could have terms like in thiscosf
unknown parameters
![Page 40: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/40.jpg)
A popular calibration tool
![Page 41: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/41.jpg)
Multi-plane calibration
Images courtesy Jean-Yves Bouguet, Intel Corp.Advantage• Only requires a plane• Don’t have to know positions/orientations• Good code available online!
– Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/ – Matlab version by Jean-Yves Bouget: http://www.vision.caltech.edu/bouguetj/calib_doc/index.html– Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/
![Page 42: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/42.jpg)
Step 1: data acquisition
![Page 43: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/43.jpg)
Step 2: specify corner order
![Page 44: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/44.jpg)
Step 3: corner extraction
![Page 45: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/45.jpg)
Step 3: corner extraction
![Page 46: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/46.jpg)
Step 4: minimize projection error
![Page 47: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/47.jpg)
Step 4: camera calibration
![Page 48: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/48.jpg)
Step 4: camera calibration
![Page 49: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/49.jpg)
Step 5: refinement
![Page 50: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/50.jpg)
Nonlinear least square methods
![Page 51: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/51.jpg)
Least square fitting
number of data pointsnumber of parameters
![Page 52: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/52.jpg)
Linear least square fittingy
t
txxtxMty 10),()(
3210),( txtxxtxM is linear, too.
model parameters
),()( iii txMyxf
residualprediction
![Page 53: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/53.jpg)
Nonlinear least square fitting
![Page 54: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/54.jpg)
Function minimization
It is very hard to solve in general. Here, we only consider a simpler problem of finding local minimum.
Least square is related to function minimization.
![Page 55: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/55.jpg)
Function minimization
![Page 56: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/56.jpg)
Quadratic functionsApproximate the function with a quadratic function within a small neighborhood
![Page 57: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/57.jpg)
Quadratic functionsA is positive definite.All eigenvalues are positive.Fall all x, xTAx>0.
negative definite
A is indefiniteA is singular
![Page 58: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/58.jpg)
Function minimization
![Page 59: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/59.jpg)
Descent methods
![Page 60: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/60.jpg)
Descent direction
![Page 61: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/61.jpg)
Steepest descent method
the decrease of F(x) per unit along h direction→
hsd is a descent direction because hTsd F’(x)=- F’(x)2<0
![Page 62: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/62.jpg)
Line search
minumum is )(
that so Find
0 h)F(x α
)('
)(0
0
0
hxFhxxF
h)F(x
T α
α
)(xF'h 0
![Page 63: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/63.jpg)
Line search
Hhhhh
T
T
0)(' 0 hxFhT α
)(xF'h 0
0
)()(
)('
0''
0'
0
Hhhhh
hxFxFh
hxFh
TT
TT
T
α
α
α
![Page 64: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/64.jpg)
Steepest descent method
isocontour gradient
![Page 65: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/65.jpg)
Steepest descent method
It has good performance in the initial stage of the iterative process. Converge very slow with a linear rate.
![Page 66: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/66.jpg)
Newton’s method
It has good performance in the final stage of the iterative process, where x is close to x*.
→
→
→→
![Page 67: Camera calibration](https://reader031.vdocuments.mx/reader031/viewer/2022020322/56814c37550346895db93ccd/html5/thumbnails/67.jpg)
Hybrid method
This needs to calculate second-order derivative which might not be available.