the trifocal tensor multiple view geometry. scene planes and homographies plane induces homography...
TRANSCRIPT
The Trifocal Tensor
Multiple View Geometry
Scene planes and homographies
plane induces homography between two views
0HFFH TT
H'eF
ve'Fe'H
Hee'
6-point algorithm
6655 Hx'xHx'xe'
x1,x2,x3,x4 in plane, x5,x6 out of plane
Compute H from x1,x2,x3,x4
He'F
Three-view geometry
The trifocal tensor
Three back-projected lines have to meet in a single line
Incidence relation provides constraint on lines
Let us derive the corresponding algebraic constraint…
Notations
iii lll
0]|[IP ]a|[AP' 4 ]b|[BP" 4
0
llPπ T
l'al'Al'P'π' T
4
TT
l"bl"Bl"P"π" T
4
TT
Incidence
iii lll
0
llPπ T
l'al'Al'P'π' T
4
TT
l"bl"Bl"P"π" T
4
TT
2rank is ]π"π'[πM ,,
e.g. is part of bundle formed by ’ and ”
l"bl'a0l"Bl'Al]m,m,m[M T
4T
4
TT
321
321 βmαmm "lβb'lαa0 T4
T4
'la T4k
"lb T
4k "lB'la'lA"lbl TT4
TT4
Incidence relation
"lBa'l'lAb"l"lB'la'lA"lbl T4
TT4
TTT4
TT4
"lba'l'lab"l T4
TT4
Tiiil "lba'l"lba'l T
4TT
4T
ii
T4
T4 babaT iii
"lT'l Tiil
The Trifocal Tensor
"lT,T,T'll 321TT
Trifocal Tensor = {T1,T2,T3}
"lT'l,"lT'l,"lT'l 3T
2T
1T
Only depends on image coordinates and is thus independent of 3D projective basis
"lT'll' TTi 'lT"ll" TT
iAlso and but no simple relation
General expression not as simple asT
4T
4 babaT iii
DOF T: 3x3x3=27 elements, 26 up to scale 3-view relations: 11x3-15=18 dof
8(=26-18) independent algebraic constraints on T(compare to 1 for F, i.e. rank-2)
Homographies induced by a plane
Line-line-line relation
Eliminate scale factor:
(up to scale)
Point-line-line relation
Point-line-point relation
note: valid for any line through x”, e.g. l”=[x”]xx”arbitrary
Point-point-point relation
note: valid for any line through x’, e.g. l’=[x’]xx’arbitrary
Overview incidence relations
Non-incident configurationincidence in image does not guarantee incidence in space
Epipolar lines
if l’ is epipolar line, then satisfied for arbitrary l”
inversely,
epipolar lines are right and left null-space of
Epipoles
With points
becomes respectively
Epipoles are intersection of right resp. left null-space of
(e=P’C and e”=P”C)
Extracting F
21Fgood choice for l” is e” (V3
Te”=0)
Computing P,P‘,P“
?
ok, but not
specifically, (no derivation)
matrix notation is impractical
Use tensor notation instead
0" jkikj
i Tllx
jkiT
Definition affine tensor
• Collection of numbers, related to coordinate choice, indexed by one or more indices
• Valency = (n+m)• Indices can be any value between 1
and the dimension of space (d(n+m)
coefficients)
Conventions
0iijbA
Einstein’s summation:(once above, once below)
ii
iji
ij bAbA
Index rule: jbA iij ,0
Contravariant indices
Covariant indices
More on tensors
• Transformations
iji
j xAx
ijji llA
(covariant)
(contravariant)
Some special tensors
• Kronecker delta
• Levi-Cevita epsilon
(valency 2 tensor)
(valency 3 tensor)
Trilinearities
Compute F and P from T
matrix notation is impractical
Use tensor notation instead
0 jkikj
i Tllx
jkiT
Definition affine tensor
• Collection of numbers, related to coordinate choice, indexed by one or more indices
• Valency = (n+m)• Indices can be any value between 1
and the dimension of space (d(n+m)
coefficients)
Conventions
0iijbA
Contraction:(once above, once below)
ii
iji
ij bAbA
Index rule: jbA iij ,0
More on tensors
• Transformations
iji
j xAx
ijji llA
(covariant)
(contravariant)
Some special tensors
• Kronecker delta
• Levi-Cevita epsilon
(valency 2 tensor)
(valency 3 tensor)
Trilinearities
Transfer: epipolar transfer
Transfer: trifocal transfer
Avoid l’=epipolar line
Transfer: trifocal transferpoint transfer
line transfer
degenerate when known lines are corresponding epipolar lines
Computation of Trifocal Tensor
• Linear method (7-point)
• Minimal method (6-point)
• Geometric error minimization method
• RANSAC method
Basic equations
Three points
Correspondence Relation #lin. indep.Eq.
4
Two points, one line
One points, two line
2
1
2Three lines
At=0 (26 equations)
(more equations)min||At|| with ||t||=1
Normalized linear algorithm
At=0
ijlmimk
ljjmk
liilk
mjjlk
mik TxxTxxTxxTxxx 0
03333 ilk
lk
iik
lk
lik TTxTxTxxx 2,1, li
Points
Lines
or
Normalization: normalize image coordinates to ~1
Normalized linear algorithm
ObjectiveGiven n7 image point correspondences across 3 images,or a least 13 lines, or a mixture of point and line corresp., compute the trifocal tensor.Algorithm(i) Find transformation matrices H,H’,H” to normalize 3 images(ii) Transform points with H and lines with H-1
(iii) Compute trifocal tensor T from At=0 (using SVD)(iv) Denormalize trifocal tensor st
r
k
t
j
sri
jki TT ˆ"H'HH 11
Internal constraints
27 coefficients 1 free scale 18 parameters 8 internal consistency constraints
(not every 3x3x3 tensor is a valid trifocal tensor!)
(constraints not easily expressed explicitly)
Trifocal Tensor satisfies all intrinsic constraintsif it corresponds to three cameras {P,P’,P”}
Maximum Likelihood Estimation
i
iiiiii ddd 222 "x̂,x"'x̂,x'x̂,x
iii x"x'x
iiiiii X"P"x̂,XP''x̂,X]0|I[x̂
data
cost function
parameterization
(24 parameters+3N)
also possibility to use Sampson error (24 parameters)
ObjectiveCompute the trifocal tensor between two images
Algorithm
(i) Interest points: Compute interest points in each image
(ii) Putative correspondences: Compute interest correspondences (and F) between 1&2 and 2&3
(iii) RANSAC robust estimation: Repeat for N samples
(a) Select at random 6 correspondences and compute T
(b) Calculate the distance d for each putative match
(c) Compute the number of inliers consistent with T (d<t)
Choose T with most inliers
(iv) Optimal estimation: re-estimate T from all inliers by minimizing ML cost function with Levenberg-Marquardt
(v) Guided matching: Determine more matches using prediction by computed T
Optionally iterate last two steps until convergence
Automatic computation of T
108 putative matches 18 outliers
88 inliers 95 final inliers
(26 samples)
(0.43)(0.23)
(0.19)
additional line matches
Matrix formulation for m-View
Consider one object point X and its m images: ixi=PiXi, i=1, …. ,m:
i.e. rank(M) < m+4 .
Laplace expansions
• The rank condition on M implies that all (m+4)*(m+4) minors of M are equal to 0.
• These can be written as sums of products of camera matrix parameters and image coordinates.
Matrix formulation
for non-trivially zero minors, one row has to be taken from each image (m).
4 additional rows left to choose
lk
jihgfedcba
000000000000000
ihgfedcba
jkl
only interesting if 2 or 3 rows from view
The three different types
1. Take the 2 remaining rows from one image block and the other two from another image block, gives the 2-view constraints.
2. Take the 2 remaining rows from one image block 1 from another and 1 from a third, gives the 3-view constraints.
3. Take 1 row from each of four different image blocks, gives the 4-view constraints.