Download - Computing F class 13
![Page 1: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/1.jpg)
Computing Fclass 13
Multiple View GeometryComp 290-089Marc Pollefeys
![Page 2: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/2.jpg)
Multiple View Geometry course schedule(subject to change)
Jan. 7, 9 Intro & motivation Projective 2D Geometry
Jan. 14, 16
(no class) Projective 2D Geometry
Jan. 21, 23
Projective 3D Geometry (no class)
Jan. 28, 30
Parameter Estimation Parameter Estimation
Feb. 4, 6 Algorithm Evaluation Camera Models
Feb. 11, 13
Camera Calibration Single View Geometry
Feb. 18, 20
Epipolar Geometry 3D reconstruction
Feb. 25, 27
Fund. Matrix Comp. Structure Comp.
Mar. 4, 6 Planes & Homographies Trifocal Tensor
Mar. 18, 20
Three View Reconstruction
Multiple View Geometry
Mar. 25, 27
MultipleView Reconstruction
Bundle adjustment
Apr. 1, 3 Auto-Calibration Papers
Apr. 8, 10
Dynamic SfM Papers
Apr. 15, 17
Cheirality Papers
Apr. 22, 24
Duality Project Demos
![Page 3: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/3.jpg)
Two-view geometry
Epipolar geometry
3D reconstruction
F-matrix comp.
Structure comp.
![Page 4: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/4.jpg)
C1
C2
l2
l1
e1
e20m m 1
T2 F
Fundamental matrix (3x3 rank 2
matrix)
1. Computable from corresponding points
2. Simplifies matching3. Allows to detect wrong
matches4. Related to calibration
Underlying structure in set of matches for rigid scenes
l2
C1m1
L1
m2
L2
M
C2
m1
m2
C1
C2
l2
l1
e1
e2
m1
L1
m2
L2
M
l2lT1
Epipolar geometry
Canonical representation:
]λe'|ve'F][[e'P' 0]|[IP T
1T
1 lPπ
1T
1T
22 lPPl
0lx 2T
2
111 xel
0x][ePPx 1x1T
1T
2T2
![Page 5: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/5.jpg)
The projective reconstruction theorem
If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent
allows reconstruction from pair of uncalibrated images!
![Page 6: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/6.jpg)
ObjectiveGiven two uncalibrated images compute (PM,P‘M,{XMi})(i.e. within similarity of original scene and cameras)
Algorithm(i) Compute projective reconstruction (P,P‘,{Xi})
(a) Compute F from xi↔x‘i(b) Compute P,P‘ from F(c) Triangulate Xi from xi↔x‘i
(ii) Rectify reconstruction from projective to metricDirect method: compute H from control points
Stratified method:(a) Affine reconstruction: compute ∞
(b) Metric reconstruction: compute IAC
ii HXXE -1
M PHP -1M HPP ii HXXM
π0|I
H
100AH
-1
1TT ωMMAA
![Page 7: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/7.jpg)
Image information provided
View relations and projective objects
3-space objects
reconstruction ambiguity
point correspondences F projective
point correspondences including vanishing points F,H∞ ∞
affine
Points correspondences and internal camera calibration
F,H∞
’
∞
∞
metric
![Page 8: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/8.jpg)
Epipolar geometry: basic equation
0Fxx'T
separate known from unknown
0'''''' 333231232221131211 fyfxffyyfyxfyfxyfxxfx
0,,,,,,,,1,,,',',',',',' T333231232221131211 fffffffffyxyyyxyxyxxx
(data) (unknowns)(linear)
0Af
0f1''''''
1'''''' 111111111111
nnnnnnnnnnnn yxyyyxyxyxxx
yxyyyxyxyxxx
![Page 9: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/9.jpg)
the singularity constraint
0Fe'T 0Fe 0detF 2Frank
T333
T222
T111
T
3
2
1
VσUVσUVσUVσ
σσ
UF
SVD from linearly computed F matrix (rank 3)
T222
T111
T2
1
VσUVσUV0
σσ
UF'
FF'-FminCompute closest rank-2 approximation
![Page 10: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/10.jpg)
![Page 11: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/11.jpg)
the minimum case – 7 point correspondences
0f1''''''
1''''''
777777777777
111111111111
yxyyyxyxyxxx
yxyyyxyxyxxx
T9x9717x7 V0,0,σ,...,σdiagUA
9x298 0]VA[V T8
T ] 000000010[Ve.g.V
1...70,)xλFF(x 21T iii
one parameter family of solutions
but F1+F2 not automatically rank 2
![Page 12: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/12.jpg)
F1 F2
F
3
F7pts
0λλλ)λFFdet( 012
23
321 aaaa
(obtain 1 or 3 solutions)
(cubic equation)
0)λIFFdet(Fdet)λFFdet( 1-12221
the minimum case – impose rank 2
Compute possible as eigenvalues of (only real solutions are potential solutions)
1-12 FF
![Page 13: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/13.jpg)
0
1´´´´´´
1´´´´´´
1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
f
f
f
f
f
f
f
f
f
yxyyyyxxxyxx
yxyyyyxxxyxx
yxyyyyxxxyxx
nnnnnnnnnnnn
~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100
!Orders of magnitude differenceBetween column of data matrix least-squares yields poor results
the NOT normalized 8-point algorithm
![Page 14: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/14.jpg)
Transform image to ~[-1,1]x[-1,1]
(0,0)
(700,500)
(700,0)
(0,500)
(1,-1)
(0,0)
(1,1)(-1,1)
(-1,-1)
1
1500
2
10700
2
Least squares yields good results (Hartley, PAMI´97)
the normalized 8-point algorithm
![Page 15: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/15.jpg)
algebraic minimization
possible to iteratively minimize algebraic distance subject to det F=0 (see book if interested)
![Page 16: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/16.jpg)
Geometric distance
Gold standard
Sampson error
Symmetric epipolar distance
![Page 17: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/17.jpg)
Gold standard
Maximum Likelihood Estimation
i
iiii dd 22 'x̂,x'x̂,x
(= least-squares for Gaussian noise)
0x̂F'x̂ subject to T
iXt],|[MP'0],|[IP Parameterize:
Initialize: normalized 8-point, (P,P‘) from F, reconstruct Xi
iiii XP'x̂,PXx̂
Minimize cost using Levenberg-Marquardt(preferably sparse LM, see book)
(overparametrized)
![Page 18: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/18.jpg)
Gold standard
Alternative, minimal parametrization (with a=1)
(note (x,y,1) and (x‘,y‘,1) are epipoles)
problems: • a=0 pick largest of a,b,c,d to fix
• epipole at infinity
pick largest of x,y,w and of x’,y’,w’
4x3x3=36 parametrizations!
reparametrize at every iteration, to be sure
![Page 19: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/19.jpg)
Zhang&Loop’s approach CVIU’01
![Page 20: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/20.jpg)
First-order geometric error (Sampson error)
ee
1TT JJ T
T
JJ
ee (one eq./pointJJT scalar)
0Fxx'T e
001
Fx'T
ix
e
22
21
2
2T2
1TT FxFxFx'Fx'JJ
T
T
JJ
ee
22
21
2
2T2
1T
T
FxFxFx'Fx'
Fxx'
(problem if some x is located at epipole)
advantage: no subsidiary variables required
![Page 21: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/21.jpg)
Symmetric epipolar error
2
221
2
2T2
1T
T
FxFx
1
Fx'Fx'
1Fxx'
i
iiii dFd2T2 x'F,xx,x'
![Page 22: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/22.jpg)
Some experiments:
![Page 23: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/23.jpg)
Some experiments:
![Page 24: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/24.jpg)
Some experiments:
![Page 25: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/25.jpg)
Some experiments:
i
iiii dFd2T2 x'F,xx,x'
(for all points!)
Residual error:
![Page 26: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/26.jpg)
Recommendations:
1. Do not use unnormalized algorithms
2. Quick and easy to implement: 8-point normalized
3. Better: enforce rank-2 constraint during minimization
4. Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation)
![Page 27: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/27.jpg)
Special case:
Enforce constraints for optimal results:Pure translation (2dof), Planar motion (6dof), Calibrated case (5dof)
![Page 28: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/28.jpg)
The envelope of epipolar lines
What happens to an epipolar line if there is noise?
Monte Carlo
n=10 n=15 n=25 n=50
l2Tll kC
![Page 29: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/29.jpg)
Other entities?
Lines give no constraint for two view geometry(but will for three and more views)
Curves and surfaces yield some constraints related to tangency
![Page 30: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/30.jpg)
Automatic computation of F
(i) Interest points(ii) Putative correspondences(iii) RANSAC (iv) Non-linear re-estimation of F(v) Guided matching(repeat (iv) and (v) until stable)
![Page 31: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/31.jpg)
• Extract feature points to relate images
• Required properties:• Well-defined
(i.e. neigboring points should all be different)
• Stable across views(i.e. same 3D point should be extracted as feature for neighboring viewpoints)
Feature points
![Page 32: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/32.jpg)
homogeneous
edge
corner
dxdyyxwyI
xI
W yIxI
,
M
M should have large eigenvalues
(e.g.Harris&Stephens´88; Shi&Tomasi´94)
MTSSD
Find points that differ as much as possible from all neighboring points
Feature = local maxima (subpixel) of F(1, 2)
Feature points
![Page 33: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/33.jpg)
Select strongest features (e.g. 1000/image)
Feature points
![Page 34: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/34.jpg)
Evaluate NCC for all features withsimilar coordinates
Keep mutual best matchesKeep mutual best matches
Still many wrong matches!Still many wrong matches!
10101010 ,,´´, e.g. hhww yyxxyx
?
Feature matching
![Page 35: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/35.jpg)
0.96 -0.40 -0.16 -0.39 0.19
-0.05 0.75 -0.47 0.51 0.72
-0.18 -0.39 0.73 0.15 -0.75
-0.27 0.49 0.16 0.79 0.21
0.08 0.50 -0.45 0.28 0.99
1 5
24
3
1 5
24
3
Gives satisfying results for small image motions
Feature example
![Page 36: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/36.jpg)
• Requirement to cope with larger variations between images• Translation, rotation, scaling• Foreshortening• Non-diffuse reflections• Illumination
geometric transformations
photometric changes
Wide-baseline matching…
![Page 37: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/37.jpg)
Wide baseline matching for two different region types
(Tuytelaars and Van Gool BMVC 2000)
Wide-baseline matching…
![Page 38: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/38.jpg)
Step 1. Extract featuresStep 2. Compute a set of potential matchesStep 3. do
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
Step 3.3 determine inliers
until (#inliers,#samples)<95%
samples#7)1(1
matches#inliers#
#inliers 90%
80%
70% 60%
50%
#samples
5 13 35 106 382
Step 4. Compute F based on all inliersStep 5. Look for additional matchesStep 6. Refine F based on all correct matches
(generate hypothesis)
(verify hypothesis)
RANSAC
![Page 39: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/39.jpg)
restrict search range to neighborhood of epipolar line (1.5 pixels)
relax disparity restriction (along epipolar line)
Finding more matches
![Page 40: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/40.jpg)
• Degenerate cases• Planar scene• Pure rotation
• No unique solution• Remaining DOF filled by noise• Use simpler model (e.g. homography)
• Model selection (Torr et al., ICCV´98, Kanatani, Akaike)
• Compare H and F according to expected residual error (compensate for model complexity)
Degenerate cases:
![Page 41: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/41.jpg)
• Absence of sufficient features (no texture)• Repeated structure ambiguity
(Schaffalitzky and Zisserman, BMVC‘98)
• Robust matcher also finds Robust matcher also finds support for wrong hypothesissupport for wrong hypothesis• solution: detect repetition solution: detect repetition
More problems:
![Page 42: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/42.jpg)
geometric relations between two views is fully
described by recovered 3x3 matrix F
two-view geometry
![Page 43: Computing F class 13](https://reader035.vdocuments.mx/reader035/viewer/2022081513/56815167550346895dbf95d4/html5/thumbnails/43.jpg)
Next class: image pair rectificationreconstructing points and lines