cs 395/495-25: spring 2003 ibmr: week 8a chapter 8... epipolar geometry: two cameras: jack tumblin...

CS 395/495-25: Spring 2003CS 395/495-25: Spring 2003

IBMR: Week 8A IBMR: Week 8A

Chapter 8... Chapter 8... Epipolar Geometry: Epipolar Geometry:

Two Cameras: Two Cameras:

Jack TumblinJack [email protected]@cs.northwestern.edu

IBMR-Related SeminarsIBMR-Related Seminars

Light Scattering ModelsLight Scattering ModelsFor Rendering Human HairFor Rendering Human Hair

Steve Marschner, Cornell University Steve Marschner, Cornell University Friday May 23 3:00pm, Rm 381, CS DeptFriday May 23 3:00pm, Rm 381, CS Dept..

RemindersReminders

• ProjA graded: Good Job! 90,95, 110ProjA graded: Good Job! 90,95, 110• ProjB graded: Good! minor H confusions.ProjB graded: Good! minor H confusions.• MidTerm graded: novel solutions MidTerm graded: novel solutions

encouraged.encouraged.• ProjC due Friday, May 16: many rec’d...ProjC due Friday, May 16: many rec’d...

• ProjD posted, due Friday May 30 ProjD posted, due Friday May 30 • (Last Week’s IAC notes revised...)(Last Week’s IAC notes revised...)• Take-Home Final Exam: Thurs June 5, due Take-Home Final Exam: Thurs June 5, due

June 11June 11

Camera Matrix P links PCamera Matrix P links P33PP22

• Basic cameraBasic camera: : x = Px = P0 0 X X wherewhere PP00 = [K | 0] = = [K | 0] =

• World-space camera:World-space camera: translate X to camera location C, then rotate:translate X to camera location C, then rotate:

x = PX = (Px = PX = (P00·R·T) X·R·T) X

• Rewrite as:Rewrite as:P = K[R | -RC]P = K[R | -RC]

• Redundant notation:Redundant notation:P = [M P = [M | p p44]]

M = KRM = KRpp44 = -K R C = -K R C

Output: Output: xx

2D Camera Image2D Camera Image Input:Input: XX

3D World Space3D World Space

zz

yy

xx

XX (world space)(world space)

zz cc

yy cc

xx cc

CC

xx(c

amer

a (c

amer

a sp

ace)

spac

e)

~~

~~

xxff ss ppxx 00

0 0 yyff ppyy 00

0 0 0 0 11 0 0

~~

Chapter 7: More One-Camera DetailsChapter 7: More One-Camera Details

Full 3x4 camera matrix Full 3x4 camera matrix PP maps maps PP33worldworld to to PP2 2

imageimage

? What does it do to basic 3D world shapes?? What does it do to basic 3D world shapes?

ForwardForward Projection: Projection:

• Line / Ray in Line / Ray in worldworld Line/Ray in Line/Ray in imageimage::– Ray in Ray in PP33 is: is: XX(() = A + ) = A + BB– Camera changes to Camera changes to PP22: : x(x() = PA + ) = PA + PB PB

ff CC

yycc

zzcc

xxcc

pp

PAPA

AA

BB

Chapter 7: More One-Camera DetailsChapter 7: More One-Camera Details

Full 3x4 camera matrix Full 3x4 camera matrix PP maps maps PP33worldworld to to PP2 2

imageimage

? What does it do to basic 3D world shapes?? What does it do to basic 3D world shapes?

BackwardBackward Projection: Projection:

• Line Line LL in in imageimage Plane Plane LL in in worldworld::

– Recall: Line L in Recall: Line L in PP2 2 (a 3-vector): (a 3-vector): L =L = [x[x1 1 xx22 x x33]]TT

– Plane Plane L L in in PP33 (a 4-vector): (a 4-vector):

LL = P = PTT··L = L =

ff CC

yycc

zzcc

xxcc

pp

LL

xx11

xx22

xx33

pp1111 p p2121 p p3131

pp1212 p p2222 p p3232

pp1313 p p2323 p p3333

pp1414 p p2424 p p3434

Cameras as ProtractorsCameras as Protractors

• World-Space Direction:World-Space Direction: D = [xD = [xcc, y, ycc, z, zcc, 0], 0]TT

• Direction from image point Direction from image point xx: : D = (KR)D = (KR)-1-1xx• Point Point C C and points and points xx11,x,x2 2 form angle form angle : : (pg 199)(pg 199)

(K(K-T-TKK-1-1)) = = = = ‘I‘Image ofmage of A Absolutebsolute C Coniconic’’

Line Line LL makes plane makes plane with normal with normal n = Kn = KT T L L (in camera coords)

coscos = = xx11T T (K(K-T-TKK-1-1)) x x22

(x(x11T T (K(K-T-TKK-1-1)) x x11) (x) (x22

TT (K (K-T-TKK-1-1) ) xx22))

CC

xx 22

xx 11

LL

dd 22

dd 11

nn


P (P ()) = = (K(K-T-TKK-1-1). ). OK.OK. Now what was Now what was again?again?

• Absolute Conic Absolute Conic =all =all imaginary pointsimaginary points on on

– Satisfies BOTH Satisfies BOTH xx112 2 + x+ x22

2 2 + x+ x332 2 = 0= 0 AND AND xx44

2 2 = 0= 0

– Finds right anglesFinds right angles-- if -- if DD11 DD22, then: , then: DD11TT· · ··DD22 = 0= 0

• Dual of absolute conic is Dual of absolute conic is Dual QuadricDual Quadric QQ**

= all= all imaginaryimaginary planesplanes toto ,, tangent to tangent to

Q*Q*

– Finds right anglesFinds right angles-- if -- if 11 22, then: , then: 11TT· · QQ**··22 = 0= 0

– can writecan write or or Q*Q* as the same matrix: as the same matrix:

1 0 0 01 0 0 00 1 0 00 1 0 00 0 1 00 0 1 00 0 0 00 0 0 0

xx11

xx22

xx33

00

xx1 1 x x22 x x3 3 00


Clever vanishing point trick: Clever vanishing point trick: • Perpendicular lines in image?Perpendicular lines in image?• Find their vanishing pts. by construction:Find their vanishing pts. by construction:• Use Use vv11

TT v v22 = 0 = 0, stack, solve for , stack, solve for = = (K(K-T-TKK-1-1))

vv11

vv22

vv33


P P = = (K(K-T-TKK-1-1)) = = = = ‘I‘Image ofmage of A Absolutebsolute C Coniconic’’

• Just Just as has a dual as has a dual QQ**, , has dual has dual ** : : * = * = -1-1 = K K = K KTT

• The dual conic The dual conic * * is the image of is the image of QQ** , so, so* = * = P (P (QQ**)) = = P( ) P( )

• Vanishing pointsVanishing points v v11,v,v22 of 2 of 2 world-space lines: world-space lines:vv11

TT v v22 = 0 = 0

• Vanishing linesVanishing lines L L11, L, L22 of 2 of 2 world-space planes: world-space planes:LL11

TT* L* L22 = 0 = 0

1 0 0 0 1 0 0 0 0 1 0 00 1 0 00 0 1 00 0 1 00 0 0 00 0 0 0

Movement Detection?Movement Detection?

• Can we do it from images only?Can we do it from images only?– 2D projective transforms often LOOK like 3-D; 2D projective transforms often LOOK like 3-D; – External cam. calib. affects all elements of External cam. calib. affects all elements of PP

• YES. Camera moved if-&-only-ifYES. Camera moved if-&-only-ifCamera-ray points (Camera-ray points (CCxxXX11,X,X22, etc…, etc…) will) will

map to LINE (not a point) in the other imagemap to LINE (not a point) in the other image

• ‘‘Epipolar LineEpipolar Line’ ==’ == l’ l’ = image of = image of L L

• ‘‘Parallax’ Parallax’ ==== x x11’’xx22’ ’ vectorvectorC’C’

CC

xx LL

XX11

XX22

xx22’’

xx11’’

l’l’

Epipolar Geometry: Chapter 8Epipolar Geometry: Chapter 8

Basic idea:Basic idea:

• Given Given ONLY imagesONLY images from 2 cameras from 2 cameras C, C’C, C’

• Different views of same objects Different views of same objects XX, , butbutwe don’t knowwe don’t know world-space points world-space points XX. .

• If we choose an If we choose an xx, , how can we find how can we find x’x’ ? ?

• How are How are x, x’x, x’ linked? linked?CC

C’C’

x’x’xx

XX


Basic idea:Basic idea:• 2 cameras located at 2 cameras located at C, C’C, C’ in world space. in world space.• Find Find ‘baseline’‘baseline’ through camera centers through camera centers C, C’C, C’• Baseline hits image planes at Baseline hits image planes at ‘epipoles’‘epipoles’

Notice baseline and XNotice baseline and Xform a plane...form a plane...

Many OTHER planesMany OTHER planes thru baseline ...thru baseline ... baselinebaseline

CCC’C’

x’x’xx

XX


Basic idea:Basic idea:• 2 cameras located at C, C’ in world space.2 cameras located at C, C’ in world space.• Find ‘baseline’ through camera centers C, C’Find ‘baseline’ through camera centers C, C’• Baseline hits image planes at ‘epipoles’Baseline hits image planes at ‘epipoles’• Family of planes thru baselineFamily of planes thru baseline

are all the are all the ‘epipolar planes’‘epipolar planes’• Image of planes = lines =Image of planes = lines =

‘epipolar lines’‘epipolar lines’• Lines intersect at Lines intersect at

epipolar pointsepipolar points in both images. in both images.CC

C’C’baselinebaseline

x’x’xx


Summary:Summary:• Connect cameras Connect cameras C, C’C, C’ with a with a baselinebaseline, which, which

hits image planes at hits image planes at epipolesepipoles e, e’. e, e’. • Chose any world pt Chose any world pt X, X, then then everything is coplanar!everything is coplanar!

epipolar planeepipolar plane includes includes image pointsimage points x, x’, and x, x’, and thesethese connect to connect to epipolesepipoles e,e’ by e,e’ by epipolar linesepipolar lines L, L’ L, L’

baselinebaselineCC

C’C’

epipolar line L

epipolar line L epipolar

line L

’

epipolar

line L

’epipolar plane epipolar plane XX

x’x’

e’e’

xx

ee

Epipolar GeometryEpipolar Geometry

Useful properties:Useful properties:• Every image point Every image point xx maps to an epipolar line maps to an epipolar line L’L’

also:also:– Epipoles Epipoles e,e’e,e’ = each camera’s view of the other = each camera’s view of the other – All epipolar linesAll epipolar lines L’ L’ pass through epipole pass through epipole e’e’– Epipolar Line Epipolar Line L’L’ is (image of the is (image of the CCXX ray...) ray...)– Epipolar Line Epipolar Line L’L’ links (image of links (image of CC) to (image of ) to (image of XX))

baselinebaselineCC

C’C’

epipolar line L


line L

’

epipolar

line L

’epipolar plane epipolar plane XX

x’x’

e’e’

xx

ee

Fundamental Matrix: Fx = L’Fundamental Matrix: Fx = L’

One Matrix Summarizes ALL of Epipolar GeometryOne Matrix Summarizes ALL of Epipolar Geometry

Fundamental Matrix Fundamental Matrix FF: 3x3, rank 2.: 3x3, rank 2.• Maps image point Maps image point xx to image point to image point x’:x’: x’ x’TT F x = 0 F x = 0

but F is only Rank 2 – given only x, F cannot find x’ for you!! but F is only Rank 2 – given only x, F cannot find x’ for you!!

• Maps image point Maps image point xx to epipolar line to epipolar line L’L’: : F x = L’F x = L’

baselinebaselineCC

C’C’

epipolar line L


line L

’

epipolar

line L

’

XX

x’x’

e’e’

xx

ee

epipolar plane epipolar plane


1) How do we find1) How do we find FF??

If we know the camera matrices If we know the camera matrices PP and and P’P’

(we almost never do), book derives: (we almost never do), book derives: (pg 224)(pg 224)

F = [e’]F = [e’] PP’ ’ PP++ ?!?!What?!?! point e’ cross product with a matrix p’p+ ?!?!?!?!What?!?! point e’ cross product with a matrix p’p+ ?!?!

baselinebaselineCC

C’C’

epipolar line L


line L

’

epipolar

line L

’

XX

x’x’

e’e’

xx


X(X(

))

(Recall: P(Recall: P+ = + = PPTT(PP(PPTT))-1-1, the pseudo-inverse), the pseudo-inverse)


F = [e’]F = [e’] PP’ ’ PP++ But what’s this? NEW TRICK:But what’s this? NEW TRICK:

– Cross Product written as matrix multiply Cross Product written as matrix multiply (pg. 554)(pg. 554) a a b =b = = = == [a] = = == [a]··b b

– Note: Note: a a b = -b b = -b a = [a]a = [a]··b = (ab = (aTT··[b][b]))TT

baselinebaselineCC

C’C’

epipolar line L


line L

’

epipolar

line L

’

XX

x’x’

e’e’

xx


X(X(

))

aa11

aa22

aa33

bb11

bb22

bb33

aa22bb33 – a – a33bb22

aa33bb11 – a – a11bb33

aa11bb22 – a – a22bb11

0 -a0 -a33 a a22

a a33 0 -a 0 -a11

-a-a22 a a11 0 0

bb11

bb22

bb33‘‘skew symmetric’ skew symmetric’ matrixmatrix


• 2) What is 2) What is FF if we DON’T know the cameras if we DON’T know the cameras P, P’,P, P’,but we DO know some corresp. point pairs (x, x’)?but we DO know some corresp. point pairs (x, x’)?– FF finds epipolar line finds epipolar line L’L’ from point from point x x : Fx = L’: Fx = L’– (Recall that if (any) point (Recall that if (any) point x’x’ is on line a is on line a L’L’, then , then x’x’TT L’= 0 L’= 0) ) – Substitute Substitute FxFx for L’: for L’: x’x’TTF x = 0F x = 0

AHA! we can find F using DLT-like method! AHA! we can find F using DLT-like method! see Chap. 10see Chap. 10

baselinebaselineCC

C’C’

epipolar line L


line L

’

epipolar

line L

’

XX

x’x’

e’e’

xx


X(X(

))

Fundamental Matrix SummaryFundamental Matrix Summary

FF is 3x3 matrix, maps is 3x3 matrix, maps PP22PP22, rank 2, 7-DOF, rank 2, 7-DOF

• If world space pt If world space pt XX image space pts. image space pts. x x and and x’x’ then then

x’x’TTF x = 0F x = 0

• Every image pt has epipolar line in the other image:Every image pt has epipolar line in the other image:

Fx = L’Fx = L’ FFTTx’ = Lx’ = L

• Baseline pierces image planes at epipoles e, e’ :Baseline pierces image planes at epipoles e, e’ :

Fe = 0Fe = 0 FFTTe’ = 0e’ = 0

(pg. 226)(pg. 226)

Fundamental Matrix SummaryFundamental Matrix Summary

FF is 3x3 matrix, maps is 3x3 matrix, maps PP22PP22, rank 2, 7-DOF, rank 2, 7-DOF

• Given camera matrices Given camera matrices PP, , P’P’, find , find FF matrix by: matrix by:

F = [e’]F = [e’] P’ PP’ P++ (recall: e’ is image of C: e’ = P’C)(recall: e’ is image of C: e’ = P’C)

• FF is unaffected by any proj. transforms done is unaffected by any proj. transforms done on BOTH camerason BOTH cameras

((PHPH, , P’HP’H) has same F matrix as () has same F matrix as (PP, , P’P’) for ) for anyany full-rank full-rank HH

(e.g. F measures camera C vs. Camera C’ only, (e.g. F measures camera C vs. Camera C’ only, no matter where you put them)no matter where you put them)

(pg. 226)(pg. 226)

Fundamental Matrix UsesFundamental Matrix Uses

Special case: camera translate only Special case: camera translate only (no rotations)(no rotations)

• Camera matrices are Camera matrices are P= K[ P= K[ I | 0 I | 0 ]], , P’ = K[P’ = K[I | t I | t ]]– where where KK is internal calib., is internal calib., tt is 3D translation vector is 3D translation vector

• F matrix simplifies to F matrix simplifies to F = [e’]F = [e’]• Epipolar lines are all parallel to direction Epipolar lines are all parallel to direction t t

• x,x’x,x’ displacement depends only on displacement depends only on tt & 3D depth & 3D depth zz::

x’ = x + (Kx’ = x + (Ktt)(1/z)(1/zcc))

ttxx

ttyy

ttzz

Fundamental Matrix UsesFundamental Matrix Uses

Can we find a camera matrix from motion + fundamental Can we find a camera matrix from motion + fundamental FF??

• Let one camera position define the world’s coords:Let one camera position define the world’s coords:

P = PP = P00 = K[ = K[ I | 0 I | 0 ]], , and other is and other is

P’ = [M | m] = K[R P’ = [M | m] = K[R | - -RC] RC] where where KK is internal calib., is internal calib., R R is rotation is rotation CC is position is position

• FF matrix simplifies to matrix simplifies to F = F = [[mm]]M M • If we know how we moved the camera (If we know how we moved the camera (R,C matricesR,C matrices) )

then find then find FF by correspondence and solve for K. by correspondence and solve for K. (pg 237) (pg 237)

• No R, C matrices? Use Essential Matrix No R, C matrices? Use Essential Matrix (pg 238)(pg 238)

~~

~~

Fundamental Matrix PropertiesFundamental Matrix Properties

Why bother with F?Why bother with F?• Can find it from image pt. correspondences onlyCan find it from image pt. correspondences only• Works even for mismatched cameras Works even for mismatched cameras

(example: 100-year time-lapse of Eiffel tower)(example: 100-year time-lapse of Eiffel tower)• Choose your own world-space coordinate system.Choose your own world-space coordinate system.

• SVD lets us recover PSVD lets us recover P00, P’ camera matrices from F, P’ camera matrices from F

(See 8.6 ‘The Essential matrix’(See 8.6 ‘The Essential matrix’pg 240pg 240))

• Complete 2-camera mapping from worldComplete 2-camera mapping from worldimageimage– 2 images + corresponding point pairs (x2 images + corresponding point pairs (xii,x’,x’ii))FF

– Let camera coords == 3D world coords, then Let camera coords == 3D world coords, then (x(xii,x’,x’ii))XXii

(pg. 226)(pg. 226)

Correspondence Problem:Correspondence Problem:

Where Computer Vision, IBMR part ways:Where Computer Vision, IBMR part ways:

Fundamental Matrix + Fundamental Matrix + Corresponding point pairs (x,x’) Corresponding point pairs (x,x’)

How can we How can we ‘blunt’ the correspondence problem?‘blunt’ the correspondence problem?

ENDEND

cs 395/495-25: spring 2003 ibmr: week 8a chapter 8... epipolar geometry: two cameras: jack tumblin...

Documents