what is visual odometry - ucsblbmedia.ece.ucsb.edu/.../lecture6-visual-odometry.pdf · visual...
TRANSCRIPT
![Page 1: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/1.jpg)
![Page 2: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/2.jpg)
What is Visual Odometry
• 视觉里程计(Visual odometry)起初主要应
用在机器人领域中,用于解决移动机器人在未知环境中的自主定位和导航问题
• 它的核心功能是分析采集的图片序列,并由此确定相机的当前位置和姿态。根据每一帧的相机姿态,可以得到整个系统的轨迹图
![Page 3: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/3.jpg)
![Page 4: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/4.jpg)
![Page 5: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/5.jpg)
![Page 6: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/6.jpg)
![Page 7: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/7.jpg)
![Page 8: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/8.jpg)
![Page 9: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/9.jpg)
![Page 10: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/10.jpg)
![Page 11: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/11.jpg)
![Page 12: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/12.jpg)
![Page 13: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/13.jpg)
![Page 14: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/14.jpg)
baseline
Epipolar plane
Epipolar holes
Epipolar lines
![Page 15: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/15.jpg)
The epipolar geometry
C,C’,x,x’ and X are coplanar
![Page 16: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/16.jpg)
The epipolar geometry
All points on p project on l and l’
![Page 17: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/17.jpg)
The epipolar geometry
Family of planes p and lines l and l’
Intersection in e and e’
![Page 18: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/18.jpg)
The epipolar geometry
epipoles e,e’
= intersection of baseline with image plane
= projection of projection center in other image
an epipolar plane = plane containing baseline (1-D family)
an epipolar line = intersection of epipolar plane with image
(always come in corresponding pairs)
![Page 19: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/19.jpg)
Example: converging cameras
![Page 20: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/20.jpg)
Example: motion parallel with image plane
![Page 21: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/21.jpg)
Example: forward motion
e
e’
![Page 22: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/22.jpg)
Essential Matrix and Fundamental Matrix
两个相机拍摄同一场景获得的左右两幅图像对应的点之间的关系可以
通过Essential矩阵或者Fundamental矩阵来表明。
![Page 23: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/23.jpg)
Matrix form of cross product
bab
aa
aa
aa
baba
baba
baba
ba
0
0
0
12
13
23
1221
3113
2332
0)(
0)(
bab
baa
![Page 24: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/24.jpg)
Calibrated Camera
T
T
vup
vupRptp
)1,','('
)1,,( with 0)](['
0' Epp
' 0 with p Ep E t R SR Essential matrix
![Page 25: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/25.jpg)
Uncalibrated Camera
0 ' pEp
scoordinate camerain ' and toingcorrespond scoordinate pixelin points ' and pppp
p'MppMp '' and 1
int
1
int
with
0'
1
int
'
int
MEMF
F pp
T
T
Fundamental matrix
![Page 26: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/26.jpg)
Properties of fundamental and
essential matrix
• Matrix is 3 x 3
• Transpose : If F is essential matrix of cameras (P, P’).
FT is essential matrix of camera (P’,P)
• Epipolar lines: Think of p and p’ as points in the projective
plane then F p is projective line in the right image.
That is l’=F p l = FT p’
• Encodes information of the extrinisic parameters only
![Page 27: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/27.jpg)
Least square approach
1|| constraint under the
) Minimize
2
2
1
F
Fp'(p i
n
i
i
We have a homogeneous system A f =0
The least square solution is smallest singular value of A,
i.e. the last column of V in SVD of A = U D VT
![Page 28: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/28.jpg)
![Page 29: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/29.jpg)
![Page 30: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/30.jpg)
![Page 31: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/31.jpg)
![Page 32: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/32.jpg)
![Page 33: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/33.jpg)
![Page 34: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/34.jpg)
![Page 35: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/35.jpg)
![Page 36: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/36.jpg)
![Page 37: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/37.jpg)
![Page 38: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/38.jpg)
![Page 39: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/39.jpg)
![Page 40: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/40.jpg)
![Page 41: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/41.jpg)
![Page 42: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/42.jpg)
![Page 43: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/43.jpg)
![Page 44: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/44.jpg)
![Page 45: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/45.jpg)
![Page 46: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/46.jpg)
![Page 47: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/47.jpg)
3D Reconstruction
• Stereo: we know the viewing geometry (extrinsic parameters) and the intrinsic parameters: Find correspondences exploiting epipolar geometry, then reconstruct
• Structure from motion (with calibrated cameras): Find correspondences, then estimate extrinsic parameters (rotation and direction of translation), then reconstruct.
• Uncalibrated cameras: Find correspondences,
Compute projection matrices (up to a projective transformation), then reconstruct up to a projective transformation.
![Page 48: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/48.jpg)
Point reconstruction
M Xx XM 'x'
![Page 49: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/49.jpg)
Geometric error
Reconstruct matches in projective frame
by minimizing the reprojection error
Non-iterative optimal solution
![Page 50: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/50.jpg)
Reconstruction for intrinsically
calibrated cameras
• Compute the essential matrix E using normalized points.
• Select M=[I|0] M’=[R|T] then E=[Tx]R
• Find T and R using SVD of E
![Page 51: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/51.jpg)
Reconstruction from uncalibrated
cameras Reconstruction problem:
given xi↔x‘i , compute M,M‘ and Xi
ii M Xx ii XMx for all i
without additional information possible
only up to projective ambiguity
![Page 52: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/52.jpg)
Projective Reconstruction
Theorem
• Assume we determine matching points xi and xi’. Then we can compute a unique Fundamental matrix F.
• The camera matrices M, M’ cannot be recovered uniquely
• Thus the reconstruction (Xi) is not unique
• There exists a projective transformation H such that
11'
2'1
12,,1,2 HMMHMMHXX ii
![Page 53: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/53.jpg)
![Page 54: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/54.jpg)
Reconstruction ambiguity:
projective
iii XHM HM Xx P
-1
P
![Page 55: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/55.jpg)
Structure from Motion
or
Simultaneous Localization and Mapping (SLAM)
or
Visual Odometry
![Page 56: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/56.jpg)
Camera calibration
• Determine camera parameters from known
3D points or calibration object(s)
1. internal or intrinsic parameters such as
focal length, optical center, aspect ratio:
what kind of camera?
2. external or extrinsic (pose)
parameters:
where is the camera?
• How can we do this?
![Page 57: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/57.jpg)
Coordinate Systems
• World Coordinate System: It’s a known
reference coordinate system with respect to
which we calibrate the camera.
• Camera Coordinate System: It’s a
coordinate system with its origin at the
optical center of the camera.
• Pixel Coordinate System
![Page 58: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/58.jpg)
Camera:Geometry Involved
• Mathematical Definition: A camera is a mapping
between a 3D world (object space) and a 2D
image.
• Calibration: The objective of calibration is to
calculate the intrinsic and/or extrinsic parameters
of a camera given a set of images taken using the
camera.
![Page 59: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/59.jpg)
Camera Models
• Perspective:
• Orthographic:
Note: We will deal only with perspective projection
z
xfx '
z
yfy '
xx ' yy '
![Page 60: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/60.jpg)
Intrinsic Parameters
• Let be the coordinates of a point in 3D. Its projection on
the image plane is given by:
),,( zyx
Pixel Square Camera, Ideal
z
yfv
z
xfu
Pixelr Rectangula Camera, Ideal
z
ylfv
z
xkfu
Center Displaced
0
0
vz
ylfv
uz
xkfu
Axes Coordinate Rotated
0
0
)sin(
)cot(
vz
yv
uz
y
z
xu
),( vu
![Page 61: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/61.jpg)
Intrinsic Parameters
KPz
p1
1
v
u
p
1
z
y
x
P
0100
0)sin(
0
0)cot(
0
0
v
u
K
Hence, the 5 intrinsic parameters of a camera are:
θ β α vu 00
![Page 62: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/62.jpg)
Extrinsic Parameters
• The camera frame ( C ) can be different from the world frame (W).
Vectoron translati3x1 t
MatrixRotation 3x3 R
1101
wc PtRP
c
c
c
c
z
y
x
P
w
w
w
w
z
y
x
P
T
T
T
r
r
r
R
3
2
1
z
y
x
t
t
t
t
Hence, we have 6 extrinsic
Parameters
![Page 63: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/63.jpg)
Perspective Projection Matrix
• M = 3x4 matrix. Taking into consideration both the intrinsic
and extrinsic parameters:
z
T
zy
TT
zyx
TTT
tr
tvtrvr
tuttrurr
tRKM
3
0302
03021
)sin()sin(
)cot()cot(
MPz
p1
Pm
Pmv
Pm
Pmu
.
.
.
.
3
2
3
1
![Page 64: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/64.jpg)
Rotation and Translation
![Page 65: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/65.jpg)
Camera matrix
• Fold intrinsic calibration matrix K and
extrinsic pose parameters (R,t) together into
a camera matrix
• M = K [R | t ]
• (put 1 in lower r.h. corner for 11 d.o.f.)
![Page 66: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/66.jpg)
Camera matrix calibration
• Directly estimate 11 unknowns in the M
matrix using known 3D points (Xi,Yi,Zi) and
measured feature positions (ui,vi)
![Page 67: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/67.jpg)
Camera matrix calibration
• Linear regression:
– Bring denominator over, solve set of (over-
determined) linear equations. How?
– Least squares (pseudo-inverse)
![Page 68: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/68.jpg)
Projective structure from motion
• Given: m images of n fixed 3D points
• xij = Pi Xj , i = 1,… , m, j = 1, … , n
• Problem: estimate m projection matrices Pi and n 3D points Xj from the mn corresponding points xij
x1j
x2j
x3j
Xj
P1
P2
P3
Slides from Lana Lazebnik
![Page 69: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/69.jpg)
Bundle adjustment • Non-linear method for refining structure and motion
• Minimizing reprojection error
2
1 1
,),(
m
i
n
j
jiijDE XPxXP
x1j
x2j
x3
j
Xj
P1
P2
P3
P1Xj
P2Xj
P3Xj
![Page 70: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/70.jpg)
Existing attempts : SLAM
• SLAM : Simultaneous Localization and Mapping
can use many different types of sensor to acquire observation data used in building the
map such as laser rangefinders, sonar sensors and cameras.
– Well-established in robotics (using a rich array of sensors)
– Demonstrated with a single hand-held camera by Davison at 2003 (Mono-
SLAM).
– Mono-SLAM was applied to
AR system at 2004.
![Page 71: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/71.jpg)
Existing attempts : Model based tracking
• Model-based tracking is
– More robust
– More accurate
– Proposed by Lepetit et. al.
at ISMAR 2003
![Page 72: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/72.jpg)
Frame by Frame SLAM • Why?
is SLAM fundamentally harder?
Time
One frame
Find features
Draw graphics
Update camera pose and entire map Many DOF
![Page 73: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/73.jpg)
Frame by Frame SLAM • SLAM
– Updating entire map every frame is so
expensive!!!
– Needs “sparse map of high-quality features”
- A. Davison
• Proposed approach
– Use dense map (of low quality features)
– Don’t update the map every frame : Keyframes
– Split the tracking and mapping into two
threads
![Page 74: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/74.jpg)
Parallel Tracking and Mapping
• Proposed method - Split the tracking and mapping into two threads
Time
One frame
Find features
Draw graphics
Update camera pose only Simple & easy
Thread #2 Mapping
Thread #1 Tracking
Update map
![Page 75: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/75.jpg)
Parallel Tracking and Mapping
Tracking thread:
• Responsible estimation of camera
pose and rendering augmented
graphics
• Must run at 30 Hz
• Make as robust and accurate as
possible
Mapping thread:
• Responsible for providing the
map
• Can take lots of time per key
frame
• Make as rich and accurate as
possible
![Page 76: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/76.jpg)
Tracking thread • Overall flow
Pre-process frame
Project points
Measure points
Update Camera Pose
Project points
Measure points
Update Camera Pose
Draw Graphics
Coarse stage Fine stage
Map
![Page 77: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/77.jpg)
Pre-process frame
• Make for pyramid levels
640x480 320x240 160x120 80x60
![Page 78: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/78.jpg)
Pre-process frame • Make for pyramid levels
• Detect Fast corners
– E. Rosten et al (ECCV 2006)
640x480 320x240 160x120 80x60
![Page 79: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/79.jpg)
Project Points
• Use motion model to update camera pose
– Constant velocity model
Vt =(Pt – Pt-1)/∇t
Estimated current Pt+1
Previous pos Pt
Previous pos Pt-1
Pt+1=Pt+∇t’(Vt)
∇t
∇t’
![Page 80: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/80.jpg)
Project Points
• Choose subset to measure
– ~ 50 biggest features for coarse stage
– 1000 randomly selected for fine stage
640x480 320x240 160x120 80x60
~50 1000
![Page 81: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/81.jpg)
Measure Points • Generate 8x8 matching template (warped
from source key-frame:map)
• Search a fixed radius around projected
position
– Use Zero-mean SSD
– Only search at Fast corner
points
![Page 82: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/82.jpg)
Update caemra pose • 6-DOF problem
– Obtain by SFM (Three-point algorithm)
?
![Page 83: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/83.jpg)
Mapping thread • Overall flow
Stereo Initialization
Wait for new key frame
Add new map points
Optimize map
Map maintenance
Tracker
![Page 84: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/84.jpg)
Stereo Initialization
• Use five-point-pose algorithm
– D. Nister et. al. 2006
• Requires a pair of frames and feature correspondences
• Provides initial map
• User input required:
– Two clicks for two key-frames
– Smooth motion for feature correspondence
![Page 85: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/85.jpg)
Wait for new key frame
• Key frames are only added if :
– There is a sufficient baseline to the other key frame
– Tracking quality is good
– Keyframe (4 level pyramid images and its corners)
• When a key frame is added :
– The mapping thread stops whatever it is doing
– All points in the map are measured in the keyframe
– New map points are found and added to the map
![Page 86: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/86.jpg)
Add new map points • Want as many map points as possible
• Check all maximal FAST corners in the key
frame :
– Check score
– Check if already in map
• Epipolar search in a neighboring key frame
• Triangulate matches and add to map
• Repeat in four image pyramid levels
![Page 87: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/87.jpg)
Optimize map
• Use batch SFM method: Bundle Adjustment
• Adjusts map point positions and key frame
poses
• Minimize reprojection error of all points in all
keyframes (or use only last N key frames)
![Page 88: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/88.jpg)
Map maintenance
• When camera is not exploring, mapping thread
has idle time
• Data association in bundle adjustment is
reversible
• Re-attempt outlier measurements
• Try measure new map features in all old key
frames
![Page 89: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/89.jpg)
Comparison to EKF-SLAM • More Accurate
• More robust
• Faster tracking
<
SLAM based AR Proposed AR
![Page 90: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/90.jpg)
System and Results • Environment
– Desktop PC (Intel Core 2 Duo 2.66 GHz)
– OS : Linux
– Language : C++
• Tracking speed
Total 19.2 ms
Key frame preparation 2.2 ms
Feature Projection 3.5 ms
Patch search 9.8 ms
Iterative pose update 3.7 ms
![Page 91: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/91.jpg)
System and Results • Mapping scalability and speed
– Practical limit
• 150 key frames
• 6000 points
– Bundle adjustment timing
Key frames 2-49 50-99 100-149
Local Bundle Adjustment 170 ms 270 ms 440 ms
Global Bundle Adjustment 380 ms 1.7 s 6.9 s
![Page 92: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/92.jpg)
Demonstration
![Page 93: What is Visual Odometry - UCSBlbmedia.ece.ucsb.edu/.../Lecture6-visual-odometry.pdf · Visual Odometry . Camera calibration • Determine camera parameters from known 3D points or](https://reader033.vdocuments.mx/reader033/viewer/2022050516/5fa040e6b0929415437ca61c/html5/thumbnails/93.jpg)
Remaining problem
• Outlier management
• Still brittle in some scenario
– Repeated texture
– Passive stereo initialization
• Occlusion problem
• Relocation problem