lecture6 stereo systems - stanford university · •pick up a window waround image 1 p=(u,v)...

Lecture 6 -Silvio Savarese 2-Feb-18

ProfessorSilvioSavareseComputationalVisionandGeometryLab

Lecture6StereoSystemsMulti-viewgeometry

Lecture 6 -Silvio Savarese 2-Feb-18

• Stereosystems• Rectification• Correspondenceproblem

• Multi-viewgeometry• TheSFMproblem• AffineSFM

Reading: [AZ] Chapter:9“Epip.Geom.andtheFundam.MatrixTransf.”[AZ] Chapter:18“Nviewcomputationalmethods”[FP] Chapters:7“Stereopsis”[FP] Chapters:8“StructurefromMotion”

Lecture6StereoSystemsMulti-viewgeometry

• Epipolar Plane • Epipoles e, e’

• Epipolar Lines• Baseline

Epipolar geometry

p’

P

p

e e’

= intersections of baseline with image planes = projections of the other camera center

O1 O2

pp’

P

Epipolar Constraint

0=¢pEpT

E = Essential Matrix(Longuet-Higgins, 1981)

[ ] RTE ×= ´

O1 O2

E =

0 −Tz TyTz 0 −Tx−Ty Tx 0

"

#

$$$$

%

&

''''

R

Essentialmatrix

[ ] RTE ×= ´

pp’

P

Epipolar Constraint

0pFpT =¢

F = Fundamental Matrix(Faugeras and Luong, 1992)

[ ] 1-´

- ¢××= KRTKF T

O1 O2

Parallel image planes

• Epipolar lines are horizontal• Epipoles go to infinity• v-coordinates are equal

p =pupv1

!

"

####

$

%

&&&&

p ' =p 'upv1

!

"

####

$

%

&&&&

O1 O2

P

e’p p’

e

x

y

z

u

v

O1 O2

Parallel image planes

E=?K1=K2 = knownHint :

x parallel to O1O2

R = I T = (T, 0, 0)

P

e’p p’

e

x

y

z

u

v

E =

0 −Tz TyTz 0 −Tx−Ty Tx 0

"

#

$$$$

%

&

''''

R =0 0 00 0 −T0 T 0

"

#

$$$

%

&

'''

Essentialmatrixforparallelimages

T = [ T 0 0 ]R = I

[ ] RTE ×= ´

O1

P

p p’

Parallel image planes

x

y

z

horizontal!What are the directions of epipolar lines?

l = E p '

l l’

u

v

e’e

u’

v’

=0 0 00 0 −T0 T 0

"

#

$$$

%

&

'''

u 'v '1

"

#

$$$

%

&

'''=

0−TT v '

"

#

$$$

%

&

'''

Parallel image planes

How are pand p’ related?

pT ⋅ E p ' = 0

P

e’p p’

e

y

z

xO1

u

v

u’

v’

Parallel image planes

( ) ( )T

0 0 0 0p E p 0 1 0 0 0 1 0

0 0 1

uu v T v u v T Tv Tv

T Tv

¢é ù æ ö æ öç ÷ ç ÷ê ú¢ ¢ ¢= Þ - = Þ - = Þ =ç ÷ ç ÷ê úç ÷ ç ÷¢ê úë û è ø è ø

How are p and p’ related?

P

e’p p’

e

y

z

O1x

⇒ v = "v

u

v

u’

v’

Parallel image planes

Rectification: making two images “parallel” Why it is useful? • Epipolar constraint ® v = v’

• New views can be synthesized by linear interpolation

P

e’p p’

e

y

z

O1x

u

v

u’

v’

Rectification:makingtwoimages“parallel”

H

Courtesy figure S. Lazebnik

Application: view morphingS. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30

Rectification

u

u

u

v

v

v

î2

î1

îS

I1

I2

Is

P

C2

Cs

C1

H1

H2

From its reflection!

Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017

Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017

Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017

Whyareparallelimagesuseful?

• Makestriangulationeasy• Makesthecorrespondenceproblemeasier

Point triangulation

O O’

P

e’p p’e

x

y

z

u

v

u

v

p =pupv1

!

"

####

$

%

&&&&

p ' =p 'upv1

!

"

####

$

%

&&&&

pu − p 'udisparity = [Eq. 1]∝B ⋅ fz

B

z

Disparity is inversely proportional to depth z!

p'upuf

B

z

O O’

P

pu − p 'u

Disparity is inversely proportional to depth z!

Computing depth

disparity = [Eq. 1]

uu

(0, pv, 1)

∝B ⋅ fz

Disparity mapshttp://vision.middlebury.edu/stereo/

Stereo pair

Disparity map / depth map

pu p’u[Eq. 1]

pu − p 'u ∝B ⋅ fz

Whyareparallelimagesuseful?

• Makestriangulationeasy• Makesthecorrespondenceproblemeasier

Correspondence problem

p’

P

p

O O’

Given a point in 3D, discover corresponding observations in left and right images [also called binocular fusion problem]

p’ belongs to l’ = FT pp belongs to l = F p’

Correspondence problem

p’

P

p

O O’

When images are rectified, this problem is much easier!

•A Cooperative Model (Marr and Poggio, 1976)

•Correlation Methods (1970--)

•Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)

Correspondence problem

[FP] Chapters: 7

p

Correlation Methods (1970--)

image 1

Where is is ?p '

Image 2

p =uv1

!

"

###

$

%

&&&

p ' =u 'v1

!

"

###

$

%

&&&

p

image 1

Where is is ?p '

Image 2

Correlation Methods (1970--)

100

uu

p =uv1

!

"

###

$

%

&&&

p ' =u 'v1

!

"

###

$

%

&&&

u

p

image 1

Where is is ?p '

Image 2

Correlation Methods (1970--)

10030 170 100

Image 2

130

What’s the problem with this?

u

• PickupawindowW around

image 1

p = (u,v )• Buildvectorw

image 2

W

u u

Window-based correlation

u

vp

• PickupawindowW around

image 1

p = (u,v )• Buildvectorw

Example:W isa3x3windowinredw isa9x1vectorw=[100,100,100,90,100,20,150,150,145]T

• SlidethewindowWalongv=inimage2andcomputew’(u)foreachu

image 2

v

100 100100

90 100 20

150 150145

• ComputethedotproductwT w’(u)foreachuandretainthemaxvalue

uu

Window-based correlation

W

u u ' = u + d = u −1

v

Window-based correlation

image 1

Example:W isa3x3windowinredw isa9x1vectorw=[100,100,100,90,100,20,150,150,145]T

image 2

uu

What’s the problem with this?

u ' = u + d = u −1

v100 100100

90 100 20

150 150145

W

u

Changes of brightness/exposure

Changes in the mean and the variance of intensity values in corresponding windows!

Normalized cross-correlation

Find u that maximizes:(w−w)T ( "w (u)− "w )(w−w) ( "w (u)− "w )

[Eq. 2]

w = w '(u) =mean value within Wlocated at ubar in image 1

mean value within W located at u in image 2

image 1 image 2

uu

u ' = u + d = u +1

v100 100100

90 100 20

150 150145

W

u

Image 1 Image 2

scanline

Example

NCC

Credit slide S. Lazebnik

u

v

– Smaller window- More detail- More noise

– Larger window- Smoother disparity maps- Less prone to noise

Window size = 3 Window size = 20

Effect of the window’s size

Credit slide S. Lazebnik

Issues

•Fore shortening effect

•Occlusions

O O’

O O’

O O’

Issues

• However, when B/z is small, small errors in measurements imply large error in estimating depth

• To reduce the effect of foreshortening and occlusions, it is desirable to have small B / z ratio!

O O’

u u’ ue’ u u’ ue’

Large B / z ratio

Small B / z ratio

• Homogeneous regions

Issues

mismatch

•Repetitive patterns

Issues

Correspondence problem is difficult!

- Occlusions- Fore shortening- Baseline trade-off- Homogeneous regions- Repetitive patterns

Apply non-local constraints to help enforce the correspondences

• Uniqueness – For any point in one image, there should be at most

one matching point in the other image

• Ordering– Corresponding points should be in the same order in

both views

• Smoothness– Disparity is typically a smooth function of x (except

in occluding boundaries)

Non-local constraints

Lecture 5 -Silvio Savarese 2-Feb-18

• Stereosystems• Rectification• Correspondenceproblem

• Multi-viewgeometry• TheSFMproblem• AffineSFM

Lecture6StereoSystemsMulti-viewgeometry

Courtesy of Oxford Visual Geometry Group

Structure from motion problem

Structure from motion problem

x1j

x2j

xmj

Xj

M1

M2

Mm

Given m images of n fixed 3D points

•xij = Mi Xj , i = 1, … , m, j = 1, … , n

From the mxn observations xij, estimate: •m projection matrices Mi

•n 3D points Xj

x1j

x2j

xmj

Xj

motionstructure

M1

M2

Mm

Structure from motion problem

Affine structure from motion(simpler problem)

Image 1

World point Xj

Image i

From the mxn observations xij, estimate: •m projection matrices Mi (affine cameras)•n 3D points Xj

xij

=

m1

m2

0 0 0 1

!

"

####

$

%

&&&&

=

m1

m2

m3

!

"

####

$

%

&&&&

X

xE = (m1X,m2X)T

x =M X=

A2 x3 b2 x101x3 1

!

"##

$

%&&

magnification

úúú

û

ù

êêê

ë

é=

3

2

1

mmm

x =M X M = A bv 1

!

"#

$

%&=

m1

m2

m3

!

"

####

$

%

&&&&

X =m1Xm2Xm3X

!

"

####

$

%

&&&&

xE = (m1Xm3X

, m2Xm3X

)T

Perspective

Affine

= A b!"

#$X = A b!

"#$

xyz1

!

"

%%%%

#

$

&&&&

=AXE+b

[Eq. 3]

M =

m1

m2

m3

!

"

####

$

%

&&&&=

m1Xm2X1

!

"

####

$

%

&&&&

XE =xyz

!

"

###

$

%

&&&

Xj

xij =AiX j +bi

Affine cameras

For the affine case (in Euclidean space)

xi j

x1j

Image i

Image 1

[Eq. 4]

2x1 2x3 2x13x1

The Affine Structure-from-Motion Problem

Given m images of n fixed points Xj we can write

Problem: estimate m matrices Ai, m matrices biand the n positions Xj from the m´n observations xij .

2m ´ n equations in 8m + 3n - 8 unknowns

How many equations and how many unknown?

N. of cameras N. of pointsxij =AiX j +bi for i = 1, …,m and j = 1, … ,n

The Affine Structure-from-Motion Problem

Two approaches:

- Algebraic approach (affine epipolar geometry; estimate F; cameras; points)

- Factorization method

Next lecture

Multiple view geometry:Affine and Perspective structure from Motion