structure from motion - university of california, san...
TRANSCRIPT
![Page 1: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/1.jpg)
CSE 252A, Fall 2019 Computer Vision I
Structure from Motion
Computer Vision I
CSE 252A
Lecture 11
![Page 2: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/2.jpg)
CSE 252A, Fall 2019 Computer Vision I
Announcements
• Homework 3 is due Nov 5, 11:59 PM
• Homework 4 will be assigned on Nov 5
• Reading:
– Chapter 8: Structure from Motion
– Optional: Multiple View Geometry in
Computer Vision, 2nd edition, Hartley and
Zisserman
![Page 3: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/3.jpg)
CSE 252A, Fall 2019 Computer Vision I
Motion
![Page 4: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/4.jpg)
CSE 252A, Fall 2019 Computer Vision I
Motion
• Some problems of motion
– Correspondence: Where have elements of the
image moved between image frames?
– Reconstruction: Given correspondences, what
is 3D geometry of scene?
– Motion segmentation: What are regions of
image corresponding to different moving
objects?
– Tracking: Where have objects moved in the
image? (related to correspondence and
segmentation)
![Page 5: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/5.jpg)
CSE 252A, Fall 2019 Computer Vision I
Structure from Motion
Locations of
points on the object
(the “structure”)
The change in rotation and translation
between the two cameras (the “motion”)
MOVING CAMERAS ARE LIKE STEREO
![Page 6: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/6.jpg)
CSE 252A, Fall 2019 Computer Vision I
Structure from Motion (SFM)
• Objective
– Given two or more images (or video frames), without knowledge of the camera poses (rotations and translations), estimate the camera poses and 3D structure of scene.
• Considerations
– Discrete motion (wide baseline) vs. continuous (infinitesimal) motion
– Calibrated vs. uncalibrated
– Two views vs. multiple views
– Orthographic (affine) vs. perspective
![Page 7: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/7.jpg)
CSE 252A, Fall 2019 Computer Vision I
Discrete Motion, Calibrated
• Consider m images of n points, how many
unknowns?
– Unknowns
• 3D Structure: 3n
• First normalized camera P = [I | 0]
– Rotations: 3(m – 1)
– Translations (to scale): 3(m – 1) – 1
• Total: 3n + 6(m – 1) – 1
– Measurements
• 2nm
• Solution when 3n + 6(m – 1) – 1 ≤ 2nm
^
![Page 8: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/8.jpg)
CSE 252A, Fall 2019 Computer Vision I
Discrete Motion, Uncalibrated
• Consider m images of n points, how many
unknowns?
– Unknowns
• 3D Structure: 3n
• Cameras (to 3D projective transformation):
11m – 15
• Total: 3n + 11m – 15
– Measurements
• 2nm
• Solution when 3n + 11m – 15 ≤ 2nm
![Page 9: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/9.jpg)
CSE 252A, Fall 2019 Computer Vision I
Two Views, Calibrated
• Input: Two images (or video frames)
• Detect feature points
• Determine feature correspondences
• Compute the essential matrix
• Retrieve the relative camera rotation and translation (to scale) from the essential matrix
• Optional: Perform dense stereo matching using recovered epipolar geometry
• Reconstruct corresponding 3D scene points (to scale)
![Page 10: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/10.jpg)
CSE 252A, Fall 2019 Computer Vision I
Two Views, Uncalibrated
• Input: Two images (or video frames)
• Detect feature points
• Determine feature correspondences
• Compute the fundamental matrix
• Retrieve the relative camera 3D projective transformation from the fundamental matrix
• Optional: Perform dense stereo matching using recovered epipolar geometry
• Reconstruct corresponding 3D scene points (to 3D projective transformation)
![Page 11: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/11.jpg)
CSE 252A, Fall 2019 Computer Vision I
Essential Matrix (Calibrated)
• Number of point correspondences and
solutions
– 5 point correspondences, up to 10 (real)
solutions
– 6 point correspondences, 1 solution
– 7 point correspondences, 1 or 3 real solutions
(and 2 or 0 complex ones)
– 8 or more point correspondences, 1 solution
![Page 12: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/12.jpg)
CSE 252A, Fall 2019 Computer Vision I
Fundamental Matrix (Uncalibrated)
• Number of point correspondences and
solutions
– 7 point correspondences, 1 or 3 real solutions
(and 2 or 0 complex ones)
– 8 or more point correspondences, 1 solution
![Page 13: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/13.jpg)
CSE 252A, Fall 2019 Computer Vision I
Mathematics
• Essential matrix
– Linear estimation (8 or more correspondences)
– Retrieval of normalized camera projection
matrices and 3D rotation and translation (to
scale)
• Fundamental matrix
– Linear estimation (8 or more correspondences)
– Retrieval of camera projection matrices and 3D
projective transformation
![Page 14: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/14.jpg)
CSE 252A, Fall 2019 Computer Vision I
Select strongest features (e.g. 1000/image)
Feature detection
![Page 15: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/15.jpg)
CSE 252A, Fall 2019 Computer Vision I
Evaluate normalized cross correlation (or sum of squared
differences) for all features with similar coordinates
Keep mutual best matches
Still many wrong matches!
( ) 10101010
,,´,́ e.g. hhww yyxxyx +−+−
?
Feature matching
![Page 16: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/16.jpg)
CSE 252A, Fall 2019 Computer Vision I
Comments• Greedy Algorithm:
– Given feature in one image, find best match in second image
irrespective of other matches
– Suitable for small motions, little rotation, small search
window
• Otherwise
– Must compare descriptor over rotation
– Cannot consider all potential pairings (way too many), so
• Manual correspondence (e.g., photogrammetry)
• Use robust outlier rejection (e.g., RANSAC)
• More descriptive features (line segments, SIFT, larger regions, color)
• Use video sequence to track, but perform SFM w/ first and last image
![Page 17: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/17.jpg)
CSE 252A, Fall 2019 Computer Vision I
Three Views
Trifocal plane
![Page 18: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/18.jpg)
CSE 252A, Fall 2019 Computer Vision I
Trifocal Tensor
• 3x3x3 tensor
• 27 elements, 18 degrees of freedom
– 33 degrees of freedom (3 camera projection
matrices) minus 15 degrees of freedom (3D
projective transformation)
• Uses tensor notation
– Einstein summation
• Retrieve camera projection matrices
![Page 19: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/19.jpg)
CSE 252A, Fall 2019 Computer Vision I
3 Points
• Point-Point-Point
![Page 20: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/20.jpg)
CSE 252A, Fall 2019 Computer Vision I
2 Points, 1 Line
• Point-Line-Point
– Note: image line must pass through
corresponding image point
![Page 21: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/21.jpg)
CSE 252A, Fall 2019 Computer Vision I
1 Point, 2 Lines
• Point-Line-Line
– Note: image lines do not need to correspond,
but must pass through corresponding image
points
![Page 22: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/22.jpg)
CSE 252A, Fall 2019 Computer Vision I
3 Lines
• Line-Line-Line
![Page 23: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/23.jpg)
CSE 252A, Fall 2019 Computer Vision I
N-View Geometry
• Reconstruction
– Bundle adjustment
• Simultaneous adjustment of parameters for all
cameras and all 3D scene points
• Minimize reprojection error in all images
• Reconstruction of cameras and 3D scene points to
similarity (calibrated) or projective (uncalibrated)
ambiguity
– Factorization (see text)
![Page 24: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/24.jpg)
CSE 252A, Fall 2019 Computer Vision I
Direct Reconstruction
Projective to Euclidean
5 (or more)
points
correspondences
![Page 25: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/25.jpg)
CSE 252A, Fall 2019 Computer Vision I
N-View Geometry
![Page 26: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/26.jpg)
CSE 252A, Fall 2019 Computer Vision I
N-View Geometry
![Page 27: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/27.jpg)
CSE 252A, Fall 2019 Computer Vision I
N-View Geometry
• Example results
![Page 28: Structure from Motion - University of California, San Diegocseweb.ucsd.edu/classes/fa19/cse252A-a/lec11.pdf · 2019-10-30 · Structure from Motion (SFM) • Objective –Given two](https://reader034.vdocuments.mx/reader034/viewer/2022043020/5f3c1d065b71a723a46deb01/html5/thumbnails/28.jpg)
CSE 252A, Fall 2019 Computer Vision I
Next Lecture
• Mid-level vision
– Grouping and model fitting
• Reading:
– Chapter 10: Grouping and Model Fitting