Computing Fclass 13

Multiple View GeometryComp 290-089Marc Pollefeys

Two-view geometry

Epipolar geometry

3D reconstruction

F-matrix comp.

Structure comp.

C1

C2

l2

l1

e1

e20m m 1

T2 F

Fundamental matrix (3x3 rank 2

matrix)

1. Computable from corresponding points

2. Simplifies matching3. Allows to detect wrong

matches4. Related to calibration

Underlying structure in set of matches for rigid scenes

l2

C1m1

L1

m2

L2

M

C2

m1

m2

C1

C2

l2

l1

e1

e2

m1

L1

m2

L2

M

l2lT1

Epipolar geometry

Canonical representation:

]λe'|ve'F][[e'P' 0]|[IP T

1T

1 lPπ

1T

1T

22 lPPl

0lx 2T

2

111 xel

0x][ePPx 1x1T

1T

2T2

The projective reconstruction theorem

If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent

allows reconstruction from pair of uncalibrated images!

ObjectiveGiven two uncalibrated images compute (PM,P‘M,{XMi})(i.e. within similarity of original scene and cameras)

Algorithm(i) Compute projective reconstruction (P,P‘,{Xi})

(a) Compute F from xi↔x‘i(b) Compute P,P‘ from F(c) Triangulate Xi from xi↔x‘i

(ii) Rectify reconstruction from projective to metricDirect method: compute H from control points

Stratified method:(a) Affine reconstruction: compute ∞

(b) Metric reconstruction: compute IAC

ii HXXE -1

M PHP -1M HPP ii HXXM

π0|I

H

100AH

-1

1TT ωMMAA

Image information provided

View relations and projective objects

3-space objects

reconstruction ambiguity

point correspondences F projective

point correspondences including vanishing points F,H∞ ∞

affine

Points correspondences and internal camera calibration

F,H∞

’

∞

∞

metric

Epipolar geometry: basic equation

0Fxx'T

separate known from unknown

0'''''' 333231232221131211 fyfxffyyfyxfyfxyfxxfx

0,,,,,,,,1,,,',',',',',' T333231232221131211 fffffffffyxyyyxyxyxxx

(data) (unknowns)(linear)

0Af

0f1''''''

1'''''' 111111111111

nnnnnnnnnnnn yxyyyxyxyxxx

yxyyyxyxyxxx

the singularity constraint

0Fe'T 0Fe 0detF 2Frank

T333

T222

T111

T

3

2

1

VσUVσUVσUVσ

σσ

UF

SVD from linearly computed F matrix (rank 3)

T222

T111

T2

1

VσUVσUV0

σσ

UF'

FF'-FminCompute closest rank-2 approximation

Frobenius distance

)()(real is

*

1 1

2 AAtraceAAtraceaA TAm

i

n

jijFnm

the minimum case – 7 point correspondences

0f1''''''

1''''''

777777777777

111111111111

yxyyyxyxyxxx

yxyyyxyxyxxx

T9x91717717x7 ]V0 ,0 ,σ,...,σ[diagUA

9x298 0]VA[V

1...70,)xλFF(x 21T iii

one parameter family of solutions

but F1+F2 not automatically rank 2

F1 F2

F

3

F7pts

0λλλ)λFFdet( 012

23

321 aaaa

(obtain 1 or 3 solutions)

(cubic equation)

0)λIFFdet(Fdet)λFFdet( 1-12221

the minimum case – impose rank 2

Compute possible as eigenvalues of (only real solutions are potential solutions)

1-12 FF

0

1´´´´´´

1´´´´´´

1´´´´´´

33

32

31

23

22

21

13

12

11

222222222222

111111111111

f

f

f

f

f

f

f

f

f

yxyyyyxxxyxx

yxyyyyxxxyxx

yxyyyyxxxyxx

nnnnnnnnnnnn

~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100

!Orders of magnitude differenceBetween column of data matrix least-squares yields poor results

the NOT normalized 8-point algorithm

Transform image to ~[-1,1]x[-1,1]

(0,0)

(700,500)

(700,0)

(0,500)

(1,-1)

(0,0)

(1,1)(-1,1)

(-1,-1)

1

1500

2

10700

2

Least squares yields good results (Hartley, PAMI´97)

the normalized 8-point algorithm

algebraic minimization

possible to iteratively minimize algebraic distance subject to det F=0 (see book if interested)

Geometric distance

Gold standard

Sampson error

Symmetric epipolar distance

Gold standard

Maximum Likelihood Estimation

i

iiii dd 22 'x̂,x'x̂,x

(= least-squares for Gaussian noise)

0x̂F'x̂ subject to T

iXt],|[MP'0],|[IP Parameterize:

Initialize: normalized 8-point, (P,P‘) from F, reconstruct Xi

iiii XP'x̂,PXx̂

Minimize cost using Levenberg-Marquardt(preferably sparse LM, see book)

(overparametrized)

Parameterization of F

Alternative, minimal parametrization (with a=1)

(note (x,y,1) and (x‘,y‘,1) are epipoles)

problems: • a=0 pick largest of a,b,c,d to fix

• epipole at infinity

pick largest of x,y,w and of x’,y’,w’

4x3x3=36 parametrizations!

reparametrize at every iteration, to be sure

First-order geometric error (Sampson error)

ee

1TT JJ T

T

JJ

ee (one eq./pointJJT scalar)

0Fxx'T e

001

Fx'T

ix

e

22

21

2

2T2

1TT FxFxFx'Fx'JJ

T

T

JJ

ee

22

21

2

2T2

1T

T

FxFxFx'Fx'

Fxx'

(problem if some x is located at epipole)

advantage: no subsidiary variables required

Symmetric epipolar error

2

221

2

2T2

1T

T

FxFx

1

Fx'Fx'

1Fxx'

i

iiii dFd2T2 x'F,xx,x'

Some experiments:

Some experiments:

Some experiments:

Some experiments:

i

iiii dFd2T2 x'F,xx,x'

(for all points!)

Residual error:

Recommendations:

1. Do not use unnormalized algorithms

2. Quick and easy to implement: 8-point normalized

3. Better: enforce rank-2 constraint during minimization

4. Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation)

Special case:

Enforce constraints for optimal results:Pure translation (2dof), Planar motion (6dof), Calibrated case (5dof)

The envelope of epipolar lines

What happens to an epipolar line if there is noise?

Monte Carlo

n=10 n=15 n=25 n=50

l2Tll kC

Other entities?

Lines give no constraint for two view geometry(but will for three and more views)

Curves and surfaces yield some constraints related to tangency

Automatic computation of F

(i) Interest points(ii) Putative correspondences(iii) RANSAC (iv) Non-linear re-estimation of F(v) Guided matching(repeat (iv) and (v) until stable)

• Extract feature points to relate images

• Required properties:• Well-defined

(i.e. neigboring points should all be different)

• Stable across views(i.e. same 3D point should be extracted as feature for neighboring viewpoints)

Feature points

homogeneous

edge

corner

dxdyyxwyI

xI

W yIxI

,

M

M should have large eigenvalues

(e.g.Harris&Stephens´88; Shi&Tomasi´94)

MTSSD

Find points that differ as much as possible from all neighboring points

Feature = local maxima (subpixel) of F(1, 2)

Feature points

Select strongest features (e.g. 1000/image)

Feature points

Evaluate Normalized Cross Correlation for all features with similar coordinates

Keep mutual best matchesKeep mutual best matches

Still many wrong matches!Still many wrong matches!

10101010 ,,´´, e.g. hhww yyxxyx

?

Feature matching

0.96 -0.40 -0.16 -0.39 0.19

-0.05 0.75 -0.47 0.51 0.72

-0.18 -0.39 0.73 0.15 -0.75

-0.27 0.49 0.16 0.79 0.21

0.08 0.50 -0.45 0.28 0.99

1 5

24

3

1 5

24

3

Gives satisfying results for small image motions

Feature example

• Requirement to cope with larger variations between images• Translation, rotation, scaling• Foreshortening• Non-diffuse reflections• Illumination

geometric transformations

photometric changes

Wide-baseline matching…

Wide baseline matching for two different region types

(Tuytelaars and Van Gool BMVC 2000)

Wide-baseline matching…

Step 1. Extract featuresStep 2. Compute a set of potential matchesStep 3. do

Step 3.1 select minimal sample (i.e. 7 matches)

Step 3.2 compute solution(s) for F

Step 3.3 determine inliers

until (#inliers,#samples)<95%

samples#7)1(1

matches#inliers#

#inliers 90%

80%

70% 60%

50%

#samples

5 13 35 106 382

Step 4. Compute F based on all inliersStep 5. Look for additional matchesStep 6. Refine F based on all correct matches

(generate hypothesis)

(verify hypothesis)

RANSAC

restrict search range to neighborhood of epipolar line (1.5 pixels)

relax disparity restriction (along epipolar line)

Finding more matches

• Degenerate cases• Planar scene• Pure rotation

• No unique solution• Remaining DOF filled by noise• Use simpler model (e.g. homography)

• Model selection (Torr et al., ICCV´98, Kanatani, Akaike)

• Compare H and F according to expected residual error (compensate for model complexity)

Degenerate cases:

• Absence of sufficient features (no texture)• Repeated structure ambiguity

(Schaffalitzky and Zisserman, BMVC‘98)

• Robust matcher also finds Robust matcher also finds support for wrong hypothesissupport for wrong hypothesis• solution: detect repetition solution: detect repetition

More problems:

geometric relations between two views is fully

described by recovered 3x3 matrix F

two-view geometry

Next class: image pair rectificationreconstructing points and lines