Lecture 5.3 Camera calibration
Thomas Opsahl
Introduction
2
• For finite projective cameras, the correspondence between points in the world and points in the image can be described by the simple model
𝑃𝑃 =𝑓𝑓𝑢𝑢 𝑠𝑠 𝑐𝑐𝑢𝑢0 𝑓𝑓𝑣𝑣 𝑐𝑐𝑣𝑣0 0 1
𝑅𝑅 𝒕𝒕
• This camera model is typically not good enough for accurate geometrical computations based on
images
Introduction
• Since light enters the camera through a lens instead of a pinhole, most cameras suffer from some kind of distortion
• This kind of distortion can be modeled and compensated for
• A radial distortion model can look like this 𝑥𝑥� = 𝑥𝑥 1 + 𝜅𝜅1 𝑥𝑥2 + 𝑦𝑦2 + 𝜅𝜅2 𝑥𝑥2 + 𝑦𝑦2 2 𝑦𝑦� = 𝑦𝑦 1 + 𝜅𝜅1 𝑥𝑥2 + 𝑦𝑦2 + 𝜅𝜅2 𝑥𝑥2 + 𝑦𝑦2 2
where 𝜅𝜅1 and 𝜅𝜅2 are the radial distortion parameters
3
X Y
Z
x
y
No radial distortion
X Y
Z
x
y
Barrel distortion
X Y
Z
x
y
Pincushion distortion
Introduction
• When we calibrate a camera we typically estimate the camera calibration matrix 𝐾𝐾 together with distortion parameters
• A model 𝐾𝐾 𝑅𝑅 𝒕𝒕 , using such an estimated 𝐾𝐾, does not in general describe the correspondence between points in the world and points in the image
• Instead it describes the correspondence between points in the world and points in a undistorted image – an image where the distortion effects have been removed
Does not satisfy 𝒖𝒖� = 𝐾𝐾 𝑅𝑅 𝑡𝑡 𝑿𝑿�
Satisfies 𝒖𝒖� = 𝐾𝐾 𝑅𝑅 𝑡𝑡 𝑿𝑿�
Image from a non-projective camera
Equivalent projective-camera image
Images: http://www.robots.ox.ac.uk/~vgg/hzbook/ 4
Undistortion
• So earlier when we estimated homographies between overlapping images, we should really have been working with undistorted images!
• How to undistort? – Matlab
[undist_img,newOrigin] = undistortImage(img,cameraParams); undistortedPoints = undistortPoints(points,cameraParams);
– OpenCV cv::undistort(img, undist_img, P, distCoeffs); cv::undistortPoints(pts,undist_pts,P,distCoeffs);
• The effect of undistortion is that we get an image or a set of points that satisfy the perspective camera
model 𝒖𝒖� = 𝑃𝑃𝑿𝑿� much better than the original image or points – So we can continue working with the simple model
5
Camera calibration
• Camera calibration is the process of estimating the matrix 𝐾𝐾 together with any distortion parameter that we use to describe radial/tangential distortion
• We have seen how 𝐾𝐾 can be found directly from the camera matrix 𝑃𝑃
• The estimation of distortion parameters can be baked into this
• One of the most common calibration algorithms was proposed by Zhegyou Zhang in the paper ’’A Flexible New Technique for Camera Calibration’’ in 2000
– OpenCV: calibrateCamera – Matlab: Camera calibration app
• This calibration algorithm makes use of multiple images of a asymmetric chessboard
6
Zhang’s method in short
• Zhang’s method requires that the calibration object is planar
• Then the 3D-2D relationship is described by a homography
[ ] [ ]1 2 3 01 1 1
1
Xu X X
Yv K K Y H Y
= = =
1 2r r r t r r t
7
𝑋𝑋𝑊𝑊
𝑌𝑌𝑊𝑊
𝑍𝑍𝑊𝑊
𝑥𝑥𝐶𝐶 𝑦𝑦𝐶𝐶
𝑧𝑧𝐶𝐶 𝐶𝐶
𝑊𝑊
Zhang’s method in short
• This observation puts 2 constraints on the intrinsic parameters due to the fact that 𝑅𝑅 is orthonormal
• Where 𝐻𝐻 = 𝒉𝒉1
𝑇𝑇 𝒉𝒉2𝑇𝑇 𝒉𝒉3
𝑇𝑇 and 𝐾𝐾−𝑇𝑇 = 𝐾𝐾𝑇𝑇 −1 = 𝐾𝐾−1 𝑇𝑇
• So for each 3D-2D correspondence we get 2 constraints on the matrix 𝐵𝐵 = 𝐾𝐾−𝑇𝑇𝐾𝐾−1
• Next step is to isolate the unknown parameters in order to compute 𝐵𝐵
11 2 1 2
1 11 1 2 2 1 1 2 2
0 0T T T
T T T T T T
K KK K K K
− −
− − − −
= =⇒
= =
r r h hr r r r h h h h
8
Zhang’s method in short
• 𝐵𝐵 can be computed directly from 𝐾𝐾
• We see that 𝐵𝐵 is symmetric, so that we can represent it by the parameter vector
𝒃𝒃 = 𝑏𝑏11 𝑏𝑏12 𝑏𝑏22 𝑏𝑏13 𝑏𝑏23 𝑏𝑏33 𝑇𝑇
( )
( ) ( )
0 02 2 2
11 12 13 20 01 0
21 22 23 2 2 2 2 2 2 2
31 32 33 22
0 0 0 00 0 0 02 2 2 2 2 2 2
1
1
1
y
u u v u v
yT
u v u v v u v v
y yy
u v u v v u v v
v s u fsf f f f f
b b b s v s u f vs sB K K b b bf f f f f f f f
b b bs v s u f v s u fv s u f v v
f f f f f f f f
− −
−−
− = = = − + −
− −− − + +
9
Zhang’s method in short
• If we denote
• Thus we have
1 1
1 2 2 1
2 2
3 1 1 3
3 2 2 3
3 3
then
i j
i j i j
i j T Tij i j ij
i j i j
i j i j
i j
h hh h h h
h hB
h h h hh h h h
h h
+
= = + +
v h h v b
1 2 1 2 12
1 1 2 2 1 1 2 2 11 22
00 00
T T T
T T T T T T
BB B
= = ⇒ ⇔ = = = −
r r h h vb
r r r r h h h h v v
10
Zhang’s method in short
• Given 𝑁𝑁 images of the planar calibration object we stack the equations to get a homogeneous system of linear equations which can be solved by SVD when 𝑁𝑁 ≥ 3
• From the estimated 𝒃𝒃 we can recover all the intrinsic parameters
• The distortion coefficients are then estimated solving a linear least-squares problem
• Finally all parameters are refined iteratively
• More details in Zhang’s paper http://research.microsoft.com/en-us/um/people/zhang/Papers/TR98-71.pdf
11
Camera calibration in practice
• OpenCV – Camera calibration tutorial – We’ll test it out in the lab
• Matlab
– App: Camera Calibrator
– This opens a new window
12
Camera calibration in practice
• Add Images and specify the size of the chessboard squares
– Chessboard detection starts
13
Camera calibration in practice
• Add Images and specify the size of the chessboard squares
– Chessboard detection starts
• Inspect the correctness of detection and choose what to estimate
– 2/3 radial distortion coeffs? – Tangential distortion? – Skew?
• Calibrate
14
Camera calibration in practice
• Add Images and specify the size of the chessboard squares
– Chessboard detection starts
• Inspect the correctness of detection and choose what to estimate
– 2/3 radial distortion coeffs? – Tangential distortion? – Skew?
• Calibrate
• Export to a Matlab variable
15
Camera calibration in practice
• Add Images and specify the size of the chessboard squares
– Chessboard detection starts
• Inspect the correctness of detection and choose what to estimate
– 2/3 radial distortion coeffs? – Tangential distortion? – Skew?
• Calibrate
• Export to a Matlab variable
16
Summary
• Calibration – Zhangs method using planar 3D-points – Matlab app – DLT method: Est 𝑃𝑃, decompose into 𝐾𝐾 𝑅𝑅 𝒕𝒕
• Undistortion
– For geometrical computations we work on undistorted images/feature points
• Additional reading – Szeliski: 6.3
• Optional reading
– A flexible new technique for camera calibration, by Z. Zhang
17