efficient large-scale stereo matchingurtasun/publications/geiger_et_al_accv... · real-time 3d...
TRANSCRIPT
www.kit.eduKIT – University of the State of Baden-Wuerttemberg andNational Laboratory of the Helmholtz Association
Andreas Geiger*, Martin Roser* and Raquel Urtasun**
Efficient Large-Scale Stereo Matching
*KARLSRUHE INSTITUTE OF TECHNOLOGY **TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO
Contents
1 Motivation and Related Work
2 Efficient Large-Scale Stereo Matching
3 Experimental Evaluation
4 Summary and Future Work
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 2/24
Contents
1 Motivation and Related Work
2 Efficient Large-Scale Stereo Matching
3 Experimental Evaluation
4 Summary and Future Work
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 3/24
Why is 3D from Stereo hard?
AmbiguitiesTextureless regions
Sensor saturationNon-Lambertian surfaces
∆z grows quadraticallyComputational burden
disparity error
focal length baseline
distance error
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 5/24
Why is 3D from Stereo hard?
AmbiguitiesTextureless regions
Sensor saturationNon-Lambertian surfaces
∆z grows quadraticallyComputational burden
disparity error
focal length baseline
distance error
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 5/24
Why is 3D from Stereo hard?
AmbiguitiesTextureless regions
Sensor saturationNon-Lambertian surfaces
∆z grows quadraticallyComputational burden
disparity error
focal length baseline
distance error
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 5/24
Related Work: Local Methods
Local MethodsWinner-takes-All
ExamplesBlock matching(Scharstein 02)Adaptive windows(Kanade 94, Yoon 06)Plane-sweep(Collins 96, Gallup 07)
ProblemsSmall matching ratiosBorder bleeding
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 6/24
Related Work: Global Methods
Global MethodsMinimize 1D/2D energy E(d) = Edata(d) + λEsmooth(d)
ExamplesGraph cuts, Belief propagation(Kolmogorov 02, Felzenszwalb 06)Variational methods(Pock 07, Zach 09)Fusion moves(Woodford 08, Bleyer 10)
ProblemsComputational and memoryrequirementsPairwise potentialscan not model planarity
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 7/24
Related Work: Seed-and-Grow
Seed-and-Grow MethodsGrow disparity components from random seeds
Examples(Cech 07)(Sara 03)
ProblemsSlanted/textureless surfacesNo dense disparity maps
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 8/24
Contents
1 Motivation and Related Work
2 Efficient Large-Scale Stereo Matching
3 Experimental Evaluation
4 Summary and Future Work
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 9/24
Idea
Assumption: rectified imagesImage pairs contain ’easy’ and ’hard’ correspondencesRobustly match ’easy’ correspondences on regular gridBuild prior on dense search space⇒ dense matching
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 10/24
Idea
Easy
Hard
Assumption: rectified imagesImage pairs contain ’easy’ and ’hard’ correspondencesRobustly match ’easy’ correspondences on regular gridBuild prior on dense search space⇒ dense matching
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 10/24
Idea
Robust Match
Assumption: rectified imagesImage pairs contain ’easy’ and ’hard’ correspondencesRobustly match ’easy’ correspondences on regular gridBuild prior on dense search space⇒ dense matching
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 10/24
Idea
Assumption: rectified imagesImage pairs contain ’easy’ and ’hard’ correspondencesRobustly match ’easy’ correspondences on regular gridBuild prior on dense search space⇒ dense matching
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 10/24
Efficient Large-Scale Stereo
NotationRobust support points S = {s1, ...,sM}with sm = (um vm dm)T
Disparity dn ∈ NObservations on = (un vn fn)T
Local image features fn
AlgorithmSplit image domain into support points S and dense pixelsAssume factorization of distribution over disparity,observations and support points into ...
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 11/24
Efficient Large-Scale Stereo
NotationRobust support points S = {s1, ...,sM}with sm = (um vm dm)T
Disparity dn ∈ NObservations on = (un vn fn)T
Local image features fn
AlgorithmSplit image domain into support points S and dense pixelsAssume factorization of distribution over disparity,observations and support points into ...
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 11/24
Efficient Large-Scale Stereo
NotationRobust support points S = {s1, ...,sM}with sm = (um vm dm)T
Disparity dn ∈ NObservations on = (un vn fn)T
Local image features fn
AlgorithmSplit image domain into support points S and dense pixelsAssume factorization of distribution over disparity,observations and support points into ...
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 11/24
Model
Left Image Graphical ModelRight Image
Support Points
Observation in left image Observations
Support Points
p(dn,o(l)n ,o(r)
n ,S) ∝ p(dn|S,o(l)n )︸ ︷︷ ︸
Prior
p(o(r)n |o
(l)n ,dn)︸ ︷︷ ︸
Likelihood
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 12/24
Model
Left Image Graphical ModelRight Image
Support Points
Observation in left image Observations
Support Points
p(dn,o(l)n ,o(r)
n ,S) ∝ p(dn|S,o(l)n )︸ ︷︷ ︸
Prior
p(o(r)n |o
(l)n ,dn)︸ ︷︷ ︸
Likelihood
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 12/24
Prior and Likelihood
Prior p(dn|S, o(l)n )
Support pt. triangulationPiecew. linear manifoldLocal extrapolation
Likelihood p(o(r)n |o
(l)n , dn)
Laplace distribution5× 5 block window3× 3 Sobel filter
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 13/24
Prior and Likelihood
Prior p(dn|S, o(l)n )
Support pt. triangulationPiecew. linear manifoldLocal extrapolation
Likelihood p(o(r)n |o
(l)n , dn)
Laplace distribution5× 5 block window3× 3 Sobel filter
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 13/24
Sampling from the model
Left image Sample mean
Right image
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 14/24
Sampling from the model
Left image Sample mean Right image
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 14/24
Contents
1 Motivation and Related Work
2 Efficient Large-Scale Stereo Matching
3 Experimental Evaluation
4 Summary and Future Work
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 15/24
Middlebury Benchmark
900 x 750 pixels, ground truth
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 16/24
Middlebury Benchmark
900 x 750 pixels, 0.4 seconds
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 16/24
Middlebury Benchmark
1300 x 1100 pixels, ground truth
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 16/24
Middlebury Benchmark
1300 x 1100 pixels, 1 second
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 16/24
Accuracy (on cones image pair)
0 0.5 1 1.5 20
5
10
15
20
Image resolution [Megapixel]
Err
or >
2px
[%]
uniform priorCech 07Felzenszwalb 06Kolmogorov 01Kostkova 03our method
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 17/24
Running times (on cones image pair)
0 0.5 1 1.5 20
500
1000
1500
2000
Image resolution [Megapixel]
Run
ning
tim
e [s
]
Kolmogorov 01Kostkova 03our method
0 0.5 1 1.5 20
10
20
30
40
50
Image resolution [Megapixel]R
unni
ng ti
me
[s]
Felzenszwalb 06Cech 07uniform priorour method
[For more details see: Geiger et al., ACCV 2010]
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 18/24
3D Reconstruction: Brussels
2 seconds[http://cvlab.epfl.ch/data/strechamvs/]
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 19/24
3D Face Reconstruction
[http://www.fujifilm.com/products/3d]
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 20/24
Contents
1 Motivation and Related Work
2 Efficient Large-Scale Stereo Matching
3 Experimental Evaluation
4 Summary and Future Work
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 22/24
Summary
Simple prior based on sparse feature matchesReduced ambiguities and run-timeTakes into account slanted surfacesReal-time 3D reconstruction of static scenes on CPUC++ / MATLAB code available at http://cvlibs.net
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 23/24
Summary
Simple prior based on sparse feature matchesReduced ambiguities and run-timeTakes into account slanted surfacesReal-time 3D reconstruction of static scenes on CPUC++ / MATLAB code available at http://cvlibs.net
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 23/24
Summary
Simple prior based on sparse feature matchesReduced ambiguities and run-timeTakes into account slanted surfacesReal-time 3D reconstruction of static scenes on CPUC++ / MATLAB code available at http://cvlibs.net
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 23/24
Summary
Simple prior based on sparse feature matchesReduced ambiguities and run-timeTakes into account slanted surfacesReal-time 3D reconstruction of static scenes on CPUC++ / MATLAB code available at http://cvlibs.net
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 23/24
Summary
Simple prior based on sparse feature matchesReduced ambiguities and run-timeTakes into account slanted surfacesReal-time 3D reconstruction of static scenes on CPUC++ / MATLAB code available at http://cvlibs.net
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 23/24
Future Work
Develop better priorsIncorporate segmentation / global reasoning on linesGPU implementation(goal: 20 fps at 1-2 megapixels)Employ as unitary potentials on global methods⇒ smaller label sets
Thank you!
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 24/24
Future Work
Develop better priorsIncorporate segmentation / global reasoning on linesGPU implementation(goal: 20 fps at 1-2 megapixels)Employ as unitary potentials on global methods⇒ smaller label sets
Thank you!
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 24/24
Future Work
Develop better priorsIncorporate segmentation / global reasoning on linesGPU implementation(goal: 20 fps at 1-2 megapixels)Employ as unitary potentials on global methods⇒ smaller label sets
Thank you!
Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 24/24