eecs 274 computer vision stereopsis. epipolar geometry trifocal tensor stereopsis: fusion and...
TRANSCRIPT
EECS 274 Computer Vision
Stereopsis
Stereopsis
• Epipolar geometry• Trifocal tensor• Stereopsis: Fusion and Reconstruction• Correlation-Based Fusion• Multi-Scale Edge Matching• Dynamic Programming• Using Three or More Cameras• Reading: FP Chapter 7, S Chapter 11
The Stanford Cart,H. Moravec, 1979.
The INRIA Mobile Robot, 1990.
Courtesy O. Faugeras and H. Moravec.
Mobile Robot Navigation
Stereo vision
• Fusion: match points observed by two or more cameras
• Reconstruction: find the pre-image of the matching points in 3D world
• Assume calibrated camera and essential matrix or trifocal tensors associated with 3 cameras are known
Reconstruction/triangulizaiton
Binocular fusion
• Epipolar plane defined by P, O, O’, p and p’
• Epipoles e, e’
• Epipolar lines l, l’
• Baseline OO’
Epipolar geometry
p’ lies on l’ where the epipolar plane intersects with image plane π’
l’ is epipolar line associated with p and intersects baseline OO’ on e’
e’ is the projection of O observed from O’
• Potential matches for p have to lie on the corresponding epipolar line l’
• Potential matches for p’ have to lie on the corresponding epipolar line l
Epipolar constraint
Epipolar Constraint: Calibrated Case
Essential Matrix(Longuet-Higgins, 1981)3 ×3 skew-symmetric
matrix: rank=2
M’=(R t)
• E is defined by 5 parameters (3 for rotation and 2 for translation)
• E p’ is the epipolar line associated with p’
• E T p is the epipolar line associated with p
• Can write as l .p = 0
• The point p lies on the epipolar line associated with the vectorE p’
Properties of essential matrix
Properties of essential matrix (cont’d)
• E e’=0 and ETe=0 (E e’=-RT[tx]e=0 )
• E is singular• E has two equal non-zero singular
values (Huang and Faugeras, 1989)
Epipolar Constraint: Small MotionsTo First-Order:
Pure translation: Focus of Expansion e
The motion field at every point in the image points to focus of expansion
Longuet-Higgins relation
Epipolar Constraint: Uncalibrated Case
Fundamental Matrix(Faugeras and Luong, 1992)are normalized image coordinate
K, K’: calibration matrices pp ˆ,ˆ
• F has rank 2 and is defined by 7 parameters
• F p’ is the epipolar line associated with p’ in the 1st image
• F T p is the epipolar line associated with p in the 2nd image
• F e’=0 and F T e=0
• F is singular
Properties of fundamental matrix
Rank-2 constraint
• F admits 7 independent parameter• Possible choice of parameterization
using e=(α,β)T and e’=(α’,β’)T and epipolar transformation
• Can be written with 4 parameters: a, b, c, d
''''''''
badcacbd
dccd
baab
F
Weak calibration
• In theory: – E can be estimated with 5 point
correspondences– F can be estimated with 7 point
correspondences– Some methods estimate E and F matrices from
a minimal number of parameters
• Estimating epipolar geometry from a redundant set of point correspondences with unknown intrinsic parameters
The Eight-Point Algorithm (Longuet-Higgins, 1981)
|F | =1.
Minimize:
under the constraint2
Homogenous system, set F33=1
Use 8 point correspondences
Least-squares minimization
• Error function:
|F | =1
Minimize:
under the constraint
factor scale :',
),'(')',(')',(
ppdppdppppe TT FFF
d(p,l): Euclidean distance between point p and line l
Non-Linear Least-Squares Approach (Luong et al., 1993)
Minimize
with respect to the coefficients of F , using an appropriate rank-2 parameterization (4 parameters instead of 8)
8 point algorithm with least-squares minimizationignores the rank 2 propertyFirst use least squares to find epipoles e and e’ that minimizes |FT e|2 and |Fe’|2
The Normalized Eight-Point Algorithm (Hartley, 1995)
• Estimation of transformation parameters suffer form poor numerical condition problem
• Center the image data at the origin, and scale it so themean squared distance between the origin and the data points is 2 pixels: q = T p , q’ = T’ p’
• Use the eight-point algorithm to compute F from thepoints q and q’
• Enforce the rank-2 constraint
• Output T F T’
T
i i i i
i
Weak calibration experiment
Trinocular Epipolar Constraints
These constraints are not independent!
Optical centers O1O2O3 defines a trifocal plane
Generally, P does not lie on trifocal plane formed
Trifocal plane intersects retinas along t1, t2, t3
Each line defines two epipoles, e.g., t2 defines e12, e32, wrt O1 and O3
Trinocular Epipolar Constraints: Transfer
Given p1 and p2 , p3 can be computedas the solution of linear equations.
Geometrically, p1 is found as the intersection of epipolar lines associated with p2 and p3
Trifocal Constraints
The set of points that project onto an image line l is the plane L that contains the line and pinhole
Point P in L is projected onto p on line l (l=(a,b,c)T)
)(
1
tR
Pp z
KM
M
Recall
P
Trifocal Constraints
All 3×3 minorsmust be zero!
Calibrated Case
Trifocal Tensorline-line-line correspondence
P
Trifocal Constraints
Calibrated Case
Given 3 point correspondences, p1, p2, p3 of the same point P, and two lines l2, l3, (passing through p2, and p3), O1p1 must intersect the line l, where the planes L2 and L3 intersect
point-line-line correspondence
011 lpT
Trifocal Constraints
Uncalibrated Case
P
Trifocal Constraints
Uncalibrated Case
Trifocal Tensor
Trifocal Constraints: 3 Points
Pick any two lines l and l through p and p .
Do it again.2 3 2 3
T( p , p , p )=01 2 3
Properties of the Trifocal Tensor
Estimating the Trifocal Tensor
• Ignore the non-linear constraints and use linear least-squaresa posteriori• Impose the constraints a posteriori
• For any matching epipolar lines, l G l = 0 • The matrices G are singular• Each triple of points 4 independent equations• Each triple of lines 2 independent equations• 4p+2l ≥ 26 need 7 triples of points or 13 triples of lines • The coefficients of tensor satisfy 8 independent constraints in the uncalibrated case (Faugeras and Mourrain, 1995) Reduce the number of independent parameters from 26 to 18
2 1 3T i
1
i
Multiple Views (Faugeras and Mourrain, 1995)
All 4 × 4 minors have zero determinants
Two Views
Epipolar Constraint
6 minors
Three Views
Trifocal Constraint
48 minors
Four Views
Quadrifocal Constraint(Triggs, 1995)
16 minors
3,...,1,,,,1,Det
l4
k3
j2
i1
lkjiijklijkl
M
M
M
M
Geometrically, the four rays must intersect in P..
Quadrifocal Tensorand Lines
Given 4 point correspondences, p1, p2, p3, p4 of the same point P, and 3 lines l2, l3, l4 (passing through p2, and p3, p4), O1p1 must intersect the line l, where the planes L2 , L3, and L4
Scale-Restraint Condition from Photogrammetry
Trinocular constraints in the presence of calibration or measurement errors
• Linear Method: find P such that(least squares)
• Non-Linear Method: find Q minimizing
Algebraic approach
Geometric approach
Stereo: Reconstruction
All epipolar lines are parallel in the rectified image plane
Retification
Disparity: d=u’-uB: baseline between O and O’
Depth: z = -B/d × f
Reconstruction from rectified images
Normalized Correlation: minimize instead.Slide the window along the epipolar line until w.w’ is maximized.
2Minimize ||w-w’||
''
'')(
ww
ww
ww
wwdC
Correlation methods (1970--)
Solution: add a second pass using disparity estimates to warpthe correlation windows, e.g. Devernay and Faugeras (1994)
Forshortening problems
[Marr, Poggio and Grimson, 1979-81]
• Edges are found by repeatedly smoothing the image and detectingthe zero crossings of the second derivative (Laplacian)• Matches at coarse scales are used to offset the search for matchesat fine scales (equivalent to eye movements)
Multiscale edge matching
One of the twoinput images
Image Laplacian
Zeros of the Laplacian
Multiscale edge matching
Multiscale edge matching
But it is not always the case..
In general the pointsare in the same orderon both epipolar lines.
Ordering constraints
Ordering constraints
points are not necessarily in orderd-b-a c’-b’-d’
[Baker and Binford, 1981]
Find the minimum-cost path going monotonicallydown and right from the top-left corner of thegraph to its bottom-right corner.
• Nodes = matched feature points (e.g., edge points).• Arcs = matched intervals along the epipolar lines.• Arc cost = discrepancy between intervals.
Dynamic programming
[Ohta and Kanade, 1985]
use DP for bothintra-scanline and inter-scanline search
use inter-scanline f or point correspondence on verticaledges
Dynamic programming
The third eye can be used for verification..
b1-a2 match is wrong as thee is no corresponding point in camera 3
Three views
[Okutami and Kanade, 1993]
Pick a reference image, and slide the correspondingwindow along the corresponding epipolar lines of allother images, using inverse depth relative to the firstimage as the search parameter.
Use the sum of correlation scores to rank matches.
More views
I1 I2 I10
Correspondence
• Extensive literature– Region based– Graph cut– Mutual information– Belief propagation– Conditional random field
• Smooth surfaces
Middlebury stereo dataset
• De facto data set with ground truth, code, and comprehensive performance evaluation
• http://vision.middlebury.edu/stereo/data/
Performance evaluation