![Page 1: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/1.jpg)
Camera Calibration
MAN-522: COMPUTER VISION
![Page 2: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/2.jpg)
Camera Calibration - Goal
• Estimate the extrinsic and intrinsic camera parameters.
f/sx f/sy
![Page 3: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/3.jpg)
Camera Calibration - How
• Using a set of known correspondences between point features
in the world (Xw, Yw, Zw) and their projections on the image
(xim, yim)
f/sx f/sy
![Page 4: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/4.jpg)
Calibration Object
• Calibration relies on one or more images of a calibration
object:
(1) A 3D object of known geometry.
(2) Located in a known position in space.
(3) Yields image features which can be located accurately.
![Page 5: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/5.jpg)
Calibration object: example
• Two orthogonal grids of
equally spaced black squares.
• Assume that the world
reference frame is centered at
the lower left corner of the
right grid, with axes parallel to
the three directions identified by
the calibration pattern.
![Page 6: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/6.jpg)
Calibration pattern: example (cont’d)
• Obtain 3D coordinates (Xw, Yw, Zw)
– Given the size of the planes, the number
of squares etc. (i.e., all known by
construction), the coordinates of each
vertex can be computed in the world
reference frame using trigonometry.
![Page 7: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/7.jpg)
Calibration pattern: example (cont’d)
• Obtain 2D coordinates (xim, yim)
– The projection of the vertices on
the image can be found by
intersecting the edge lines of the
corresponding square sides (or
through corner detection).
![Page 8: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/8.jpg)
Problem Statement
Compute the extrinsic and intrinsic camera
parameters from N corresponding pairs of points:
and (xim_i , yim_i ), i = 1, . . . , N. ( , , )W W W
i i iX Y Z
• Very well studied problem.
• There exist many different methods for camera calibration.
![Page 9: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/9.jpg)
Methods
(1) Indirect camera calibration
(1.1) Estimate the elements of the projection matrix.
(1.2) If needed, compute the intrinsic/extrinsic camera
parameters from the entries of the projection matrix.
![Page 10: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/10.jpg)
Methods (cont’d)
(2) Direct camera calibration
Direct recovery of the intrinsic and extrinsic camera
parameters.
![Page 11: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/11.jpg)
Method 1: Indirect Camera Calibration
• Review of basic equations
Note: replaced (xim,yim) with (x,y) for simplicity.
![Page 12: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/12.jpg)
(Method 1) Step 1: solve for mij’s
• M has 11 independent entries.
– e.g., divide every entry by m11
• Need at least 11 equations for computing M.
• Need at least 6 world-image point correspondences.
![Page 13: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/13.jpg)
(Method 1) Step 1: solve for mij’s
• Each 3D-2D correspondence gives rise to two equations:
( , , ) ( , )W W W
i i i i iX Y Z x y
![Page 14: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/14.jpg)
(Method 1) Step 1: solve for mij’s
• This leads to a homogeneous system of equations:
N x 12 matrix
![Page 15: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/15.jpg)
(Method 1) Step 1: solve for mij’s
![Page 16: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/16.jpg)
(Method 1) Step 2: find intrinsic/extrinsic parameters
![Page 17: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/17.jpg)
(Method 1) Step 2: find intrinsic/extrinsic parameters
• Let’s define the following vectors:
![Page 18: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/18.jpg)
• The solutions are as follows (see book chapter for
details):
• The rest parameters are easily computed ....
(Method 1) Step 2: find intrinsic/extrinsic parameters
![Page 19: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/19.jpg)
Method 2: Direct Camera Calibration
• Review of basic equations
– From world coordinates to camera coordinates
– For simplicity, we will replace -T’ with T
– Warning: this is NOT the same T as before!
Pc=RPw+T
![Page 20: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/20.jpg)
Method 2: Direct Camera Calibration (cont’d)
• Review of basic equations
– From camera coordinates to pixel coordinates:
– Relating world coordinates to pixel coordinates:
![Page 21: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/21.jpg)
Method 2: Direct Parameter Calibration
• Intrinsic parameters
– Intrinsic parameters f, sx, sy, ox, and oy are not independent.
– Define the following four independent parameters:
![Page 22: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/22.jpg)
Method 2: Main Steps
(1) Assuming that ox and oy are known,
estimate all other parameters.
(2) Estimate ox and oy
![Page 23: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/23.jpg)
(Method 2) Step 1: estimate f x , α, R, and T
• To simplify notation, set (xim- ox , yim- oy) = (x, y)
• Combining the equations above (i.e., same denominator),
we have:
![Page 24: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/24.jpg)
(Method 2) Step 1: estimate f x , α, R, and T
(cont’d)
• Each pair of corresponding points must satisfy the previous
equation:
( , , ) ( , )W W W
i i i i iX Y Z x y
divide by f y and re-arrange terms:
![Page 25: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/25.jpg)
(Method 2) Step 1: estimate f x , α, R, and T
(cont’d)
where
we obtain the following equation:
![Page 26: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/26.jpg)
(Method 2) Step 1: estimate f x , α, R, and T
(cont’d)
• Assuming N correspondences leads to a homogeneous
system :
N x 8 matrix
![Page 27: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/27.jpg)
(Method 2) Step 1: estimate f x , α, R, and T
(cont’d)
![Page 28: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/28.jpg)
(Method 2) Step 1: estimate f x , α, R, and T
(cont’d)
• Determine α and | γ |
![Page 29: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/29.jpg)
(Method 2) Step 1: estimate f x , α, R, and T
(cont’d)
• Determine r21, r22, r23, r11, r12, r13, Ty, Tx
(up to an unknown common sign)
![Page 30: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/30.jpg)
(Method 2) Step 1: estimate f x , α, R, and T
(cont’d)
• Determine r31, r32, r33
– Can be estimated as the cross product of R1 and R2:
– The sign of R3 is already fixed (the entries of R3 remain
unchanged if the signs of all the entries of R1 and R2 are
reversed).
• We have estimated R call the estimate
![Page 31: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/31.jpg)
(Method 2) Step 1: estimate f x , α, R, and T
(cont’d)
• Ensure the orthogonality of R
– The computation of R does not take into account explicitly
the orthogonality constraints.
– The estimate of R cannot be expected to be orthogonal:
– Enforce orthogonality on using SVD:
– Replace D with I:
![Page 32: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/32.jpg)
(Method 2) Step 1: estimate f x , α, R, and T
(cont’d)
• Determine the sign of γ
– Consider the following equations again:
![Page 33: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/33.jpg)
(Method 2) Step 1: estimate f x , α, R, and T
(cont’d)
![Page 34: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/34.jpg)
(Method 2) Step 1: estimate f x , α, R, and T
(cont’d)
• Determine Tz and fx
– Consider the equation:
– Let’s rewrite it in the form:
or xTz+fx(r11Xw+r12Yw+r13Zw+Tx) = -x(r31Xw+r32Yw+r33Zw)
![Page 35: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/35.jpg)
(Method 2) Step 1: estimate f x , α, R, and T
(cont’d)
– We can obtain Tz and fx by solving a system of equations like
the above, written for N points:
Using SVD, the (least-squares) solution is:
![Page 36: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/36.jpg)
(Method 2) Step 1: estimate f x , α, R, and T
(cont’d)
• Determine fy:
![Page 37: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/37.jpg)
(Method 2) Step 2: estimate ox and oy
• The computation of ox and oy is based on the following
theorem:
Orthocenter Theorem: Let T be the triangle on the image
plane defined by the three vanishing points of three
mutually orthogonal sets of parallel lines in space. Then,
(ox , oy) is the orthocenter of T.
![Page 38: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/38.jpg)
(Method 2) Step 2: estimate ox and oy (cont’d)
• We can use the same calibration pattern to compute
three vanishing points (use three pairs of parallel lines
defined by the sides of the planes).
• None of the three mutually
orthogonal directions should
not be near parallel to the
image plane!
![Page 39: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/39.jpg)
Comments
• To improve the accuracy of camera calibration, it
is a good idea to estimate the parameters several
times (i.e., using different images) and average the
results.
• Localization errors
– The precision of calibration depends on how accurately
the world and image points are located.
– Studying how localization errors "propagate" to the
estimates of the camera parameters is very important.
![Page 40: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/40.jpg)
Comments (cont’d)
• In theory, direct and indirect camera calibration
should produce the same results.
• In practice, we obtain different solutions due to
different error propagations.
• Indirect camera calibration is simpler and should be
preferred when we do not need to compute the
intrinsic/extrinsic camera parameters explicitly.
![Page 41: Image Processing Fundamentals - IIT Roorkee · 2020. 1. 7. · Calibration Object •Calibration relies on one or more images of a calibration object: (1) A 3D object of known geometry](https://reader036.vdocuments.mx/reader036/viewer/2022071415/611005dda31ed316fa22eb52/html5/thumbnails/41.jpg)
How should we estimate the accuracy
of a calibration algorithm?
• Project known 3D points on the image
• Compare their projections with the corresponding
pixel coordinates of the points.
• Repeat for many points and estimate “re-projection”
error!