multi view geometry (spring '08)ocw.snu.ac.kr/sites/default/files/note/2772.pdf · 2018. 1. 30. ·...

Prof. Kyoung Mu LeeDept. of EECS, Seoul National University

Introduction 1

Multi View Geometry (Spring '08)

Introduction

K. M. Lee, EECS, SNU

Introduction 2


Course Objectives

• To understand the geometric relations between multiple views of scenes.

• To understand how to calibrate cameras• To understand how to reconstruct 3D structure information

using views• To understand the general principles of parameter

estimation.


Introduction 3


Materials

• Text book;“Multiple View Geometry in Computer

Vision” 2nd ed. by Richard Hartley and Andrew Zisserman, Cambridge University Press

• Reference:“An Invitation to 3-D Vision”, Yi Ma et

al., Springer“The Geometry from Multiple Images”,by Olivier Faugeras and Quan-Tuan Luong, MIT Press“Introductory Techniques for 3-D Computer Vision”, E. Trucco and A. Verri, Prentice Hall


Introduction 4


Related Links

• Course Home Page: http://cv.snu.ac.kr• Related Links

The Computer Vision HomepageCVonline: Compendium of Computer VisionAnnotated Computer Vision BibliographyUSC Computer Vision BibliographyThe Digital Michelangelo ProjectIBM's Pieta ProjectOpen Directory Project CV links Matlab HomepageMatlab PrimerOnline Tutorial by Pollefey: http://www.esat.kuleuven.ac.be/~pollefey/tutorial/


Introduction 5


Grading

• Grading:Participation : 20%Homework Assignments & Quiz: 50% Final Projects: 30%

• Computer Assignments:There will be 3-4 HWs (computer projects) during the semester. The grade of the projects will be based on the report, which should include the problem statement, your approaches, implement detail, experimental results and discussion. Computer projects can be done by using MATLAB or C++.

• Term Project : Each student has to turn in a term project. This project is due one week after the end of the class. The term project must be done INDIVIDUALLY .


Introduction 6


Grading

• Academic Integrity: Every student should submit only their own original work, or where applicable, to cite its origin, for the written and programming assignments. Programs shall not be copied. Standard and publicly available libraries may be used after seeking consent of the course instructor or TA.


Introduction 7


Content

• Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation.

• Single View: Camera model, Calibration, Single View Geometry.

• Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies.

• Three Views: Trifocal Tensor, Computing T.• More Views: N-Linearities, Multiple view reconstruction,

Bundle adjustment, auto-calibration, Dynamic SfM, Cheirality, Duality


Introduction 8


Overview

• Quick Overview (by Marc pollefey)


Introduction 9


Background

La reproduction interdite (Reproduction Prohibited), 1937, René Magritte.


Introduction 10


• Points, lines & conics

• Transformations

• Cross-ratio and invariants

Projective 2D Geometry


Introduction 11


Projective 3D Geometry

• Points, lines, planes and quadrics

• Transformations

• П∞, ω∞ and Ω ∞


Introduction 12


Estimation

How to compute a geometric relation from correspondences

Linear (normalized) Non-linear and Maximum Likelihood EstimationRobust Estimator (RANSAC, LMeds)


Introduction 13


Evaluation and error analysis

How to evaluate the accuracyBounds on performance

Covariance propagation & Monte-Carlo estimation

residual

error

( )+= − JΣJΣ XP 1T


Introduction 14


Single-View Geometry

The Cyclops, c. 1914, Odilon Redon


Introduction 15


Camera Models

X.xλor

110t

11

1

11λ

3

PR

=

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

⎥⎦

⎤⎢⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

ZYX

pfpf

yx

y

x

T

pinhole (perspective) camera model

affine cameras

pushbroom camera, etc.


Introduction 16


Camera Calibration

• How to compute camera matrix P(normalized) linear, MLE,…

• How to correct Radial lens distortion


Introduction 17


More Single-View Geometry

• Projective cameras and planes, lines, conics and quadrics.

• Camera center and camera rotation

• Camera calibration and vanishing points, calibrating conic and the IAC

** CPPQ =T

coneQCPP =T


Introduction 18


Two-View Geometry

The Birth of Venus (detail), c. 1485, Sandro Botticelli


Introduction 19


Epipolar Geometry

Fundamental matrix Essential matrix

0xx' =FT ( ) 0x̂[t]'x̂ =×RT'PP,F → 'PP,E →

• What is Epipolar geometry?• What are Fundamental matrix and Essential matrix


Introduction 20


Two-View Reconstruction


• How to determine the Fundamental matrix F

Introduction 21


Epipolar Geometry Computation

(normalized) linear:

minimal:

MLE:

RANSAC… and automated two view matching

0xx' =FT

( ) 0λdet 21 =+ FF0).1,,,',',',',','( =fyxyyyxyxyxxx

( ) ( )

0x̂'x̂for which

'x̂,x'x̂,x 2i

2

=

+∑

ii

iiii dd

FT


Introduction 22


Image Rectification

Warp images to simplify epipolar geometryAlign epipolar lines


Introduction 23


Structure Computation

• How to recover 3D features• Points: Linear, optimal, direct optimal

• Also lines and vanishing points


Introduction 24


Three-View Geometry



Introduction 25


Trifocal Tensor

• Relationship between three view correspondence.


Introduction 26


Three View Reconstruction

• (normalized) linear• minimal (6 points)• MLE (Gold Standard)


Introduction 27


Multiple-View Geometry



Introduction 28


Multiple View Geometry

wxyzpqrs

lszkryjqxipwiiii Qxxxx 0=′′′′′′ εεεε

Quadrifocal tensor

wxyzpqrs

srqp Qllll 0=′′′′′′ 81 parameters, but only 29 DOF!

• Relationship between multiple view correspondence.


Introduction 29


Multiple View Reconstruction

• Affine factorization• Projective factorization

• Sequential reconstruction

[ ]nmn

mm

n

n

XXX

P

PP

xxx

xxxxxx

21

m

2

1

21

222

21

112

11

LM

L

MOMM

L

L

⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡

=

⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡


Introduction 30


Bundle Adjustment

Maximum Likelyhood Estimation for complete structure and motion

U1

U2

U3

WT

W

V

P1 P2 P3 M

=J == JJN T

12xm 3xn(in general

much larger)

( )( )∑∑= =

m

k

n

iikd

1 1

2

ki X̂P̂,x


Introduction 31


Bundle adjustment

No bundle adjustment

Bundle adjustment needed to avoid drift of virtual objectthroughout sequence

Bundle adjustment (including radial distortion)


• How to determine camera calibration matrix K directly from multiple uncalibrated images

Introduction 32


ω*Ω*

projection

constraints

Tii

Tiii Ωω KKPP ==

∗∗

Auto-calibration

Tijiijj ∞

∗∞

∗ = HH ωω


Introduction 33


Applications

• 3D modeling• Augmented Reality• Image-based rendering

• http://www.robots.ox.ac.uk/~vanguard/examples.html


Introduction 34


Applications


Applications

Scene reconstruction

Scene reconstruction

Photo Explorer

Photo Explorer

Input photographs Relative camera positions and orientations

Point cloud

Sparse correspondence


• Reading assignment:Read Ch. 1 & Ch 2.

/ColorImageDict > /JPEG2000ColorACSImageDict > /JPEG2000ColorImageDict > /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 300 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict > /GrayImageDict > /JPEG2000GrayACSImageDict > /JPEG2000GrayImageDict > /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 1200 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict > /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile () /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /False

/Description > /Namespace [ (Adobe) (Common) (1.0) ] /OtherNamespaces [ > /FormElements false /GenerateStructure false /IncludeBookmarks false /IncludeHyperlinks false /IncludeInteractive false /IncludeLayers false /IncludeProfiles false /MultimediaHandling /UseObjectSettings /Namespace [ (Adobe) (CreativeSuite) (2.0) ] /PDFXOutputIntentProfileSelector /DocumentCMYK /PreserveEditing true /UntaggedCMYKHandling /LeaveUntagged /UntaggedRGBHandling /UseDocumentProfile /UseDocumentBleed false >> ]>> setdistillerparams> setpagedevice

multi view geometry (spring '08)ocw.snu.ac.kr/sites/default/files/note/2772.pdf · 2018. 1. 30. ·...

Documents