a non-local cost aggregation method for stereo matching
DESCRIPTION
A Non-local Cost Aggregation Method for Stereo Matching. Qingxiong Yang City University of Hong Kong 2012 IEEE Conference on Computer Vision and Pattern Recognition. Outilne. Introduction Related Works Method Experimental Results Conclusion. Introduction _________________________. - PowerPoint PPT PresentationTRANSCRIPT
A Non-local Cost Aggregation Method for Stereo Matching
Qingxiong YangCity University of Hong Kong
2012 IEEE Conference on Computer Vision and Pattern Recognition 1
Outilne• Introduction• Related Works• Method• Experimental Results• Conclusion
2
Introduction_________________________
3
Introduction• Goal : Get fast and accurate disparity map.• Solution : Non-local cost aggregation + MST• Advantage : Better in low textures region Low complexity
4
Related Works_________________________
5
Related Works
6
1 •Matching cost computation
2 •Cost (support) aggregation
3 •Disparity computation and optimization
4 •Disparity refinement
[21] D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.International Journal of Computer Vision (IJCV), 47:7–42, 2002.
Related WorksLocal methods• 1=>2=>3• A local support region
with winner take all• Implicit smoothness• Fast but inaccurate.
Global methods• 1(=>2)=>3• Energy minimization
process (GC,BP,DP,Cooperative)• Per-processing• Explicit smoothness• Accurate but slow
7
Comparison (Rank in Middleburry)Real time Non-Real Method
1 - - -
2 O (no Seg.) Cross-based Aggregation>>Scanline Optimization
3 O Mean-shift >> BP>>Self-adapting4 O Mean-shift >> Cooperative Optimization5 - - -
6 O Mean-shift>>Color-weighted>>BP7 O (no Seg.) Seed Detection>>Scanline Propagation8 O Mean-shift>>BP9 O Mean-shift(Region-based)>>B-spline10 O Up-sample>>Bilateral Filter>>BP11 - - -
12 O (No Seg.)>>Convex Relaxation>>regularization13 O Mean-shift>>Image Warping>>BP
Reference(1/2)
C. Shi, G. Wang, X. Pei, H. Bei, and X. Lin. High-accuracy stereo matching based on adaptive ground control points. Submitted to IEEE TIP 2012
X. Mei, X. Sun, M. Zhou, S. Jiao, H. Wang, and X. Zhang. On building an accurate stereo matching system on graphics hardware. GPUCV 2011.
A. Klaus, M. Sormann and K. Karner. Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. ICPR 2006.
Z. Wang and Z. Zheng. A region based stereo matching algorithm using cooperative optimization. CVPR 2008.
Anonymous. A dense stereo matching with reliability aggregation and propagation. CVPR 2012 submission 1170.
Q. Yang, L. Wang, R. Yang, H. Stewénius, and D. Nistér. Stereo matching with color-weighted correlation, hierarchical belief propagation and occlusion handling. IEEE TPAMI 2009
9
Reference(2/2)
X. Sun, X. Mei, S. Jiao, M. Zhou, and H. Wang. Stereo matching with reliable disparity propagation. 3DIMPVT 2011.
L. Xu and J. Jia. Stereo matching: an outlier confidence approach. ECCV 2008.
M. Bleyer, C. Rother, and P. Kohli. Surface stereo with soft segmentation. CVPR 2010.
Q. Yang, R. Yang, J. Davis, and D. Nistér. Spatial-depth super resolution for range images. CVPR 2007.
Y. Mizukami, K. Okada, A. Nomura, S. Nakanishi, and K. Tadamura. Sub-pixel disparity search for binocular stereo vision. ICPR 2012 submission 1439.
S. Zhu, L. Zhang, and H. Jin. A locally linear regression model for boundary preserving regularization in stereo matching. ECCV 2012.
M. Bleyer, M. Gelautz, C. Rother, and C. Rhemann. A stereo approach that handles the matting problem via image warping. CVPR 2009.
10
Method_________________________
11
Method
12
1 •Matching cost computation - Bilateral Filter
2 •Cost aggregation - MST
3 •Disparity computation and optimization
4 •Disparity refinement - Median Filter
Bilateral Filter• Every sample is replaced by a weighted average of its
neighbors.• These weights reflect two forces• How close are the neighbor and the center sample• How similar are the neighbor and the center sample
• Edge-preserving and noise reducing smoothing filter
13
Bilateral Filter
14
qp
Bilateral Filter
15
Center Sample : p
Neighborhood : q
Bilateral Filter
16
Total Distance
Bilateral Filter
17
Gaussian wieght Bilateral wieghtOriginal image
• Kruskal's Algorithm• Scan all edges increasing weight order, if an edge is
safe, add it to F. 4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
Minimum Spanning Tree
18
PPT By Jonathan Davis
Orginal Graph
4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
1
4
2
5
2
5
4
3
4
4
10
1
6
3
3
2
A AB D
B B
B
C D
J C
C
E
F
D
D H
J E G
F FG I
G GI J
H J JI19
4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
2
5
2
5
4
3
4
10
1
6
3
3
2
B
B
D
J
C
C
E
F
D
D H
J
E G
F
F
G
I
G
G
I
J
H J
JI
1A D
4B C
4A B
Sort Edges (in reality they are placed in a priority queue - not sorted - but sorting them
makes the algorithm easier to visualize)
20
4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
2
5
2
5
4
3
4
10
1
6
3
3
2
B
B
D
J
C
C
E
F
D
D H
J
E G
F
F
G
I
G
G
I
J
H J
JI
1A D
4B C
4A B
Add Edge
21
4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
2
5
2
5
4
3
4
10
1
6
3
3
2
B
B
D
J
C
C
E
F
D
D H
J
E G
F
F
G
I
G
G
I
J
H J
JI
1A D
4B C
4A B
Add Edge
22
4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
2
5
2
5
4
3
4
10
1
6
3
3
2
B
B
D
J
C
C
E
F
D
D H
J
E G
F
F
G
I
G
G
I
J
H J
JI
1A D
4B C
4A B
Add Edge
23
4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
2
5
2
5
4
3
4
10
1
6
3
3
2
B
B
D
J
C
C
E
F
D
D H
J
E G
F
F
G
I
G
G
I
J
H J
JI
1A D
4B C
4A B
Add Edge
24
4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
2
5
2
5
4
3
4
10
1
6
3
3
2
B
B
D
J
C
C
E
F
D
D H
J
E G
F
F
G
I
G
G
I
J
H J
JI
1A D
4B C
4A B
Add Edge
25
4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
2
5
2
5
4
3
4
10
1
6
3
3
2
B
B
D
J
C
C
E
F
D
D H
J
E G
F
F
G
I
G
G
I
J
H J
JI
1A D
4B C
4A B
Cycle
Don’t Add Edge
26
4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
2
5
2
5
4
3
4
10
1
6
3
3
2
B
B
D
J
C
C
E
F
D
D H
J
E G
F
F
G
I
G
G
I
J
H J
JI
1A D
4B C
4A B
Add Edge
27
4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
2
5
2
5
4
3
4
10
1
6
3
3
2
B
B
D
J
C
C
E
F
D
D H
J
E G
F
F
G
I
G
G
I
J
H J
JI
1A D
4B C
4A B
Add Edge
28
4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
2
5
2
5
4
3
4
10
1
6
3
3
2
B
B
D
J
C
C
E
F
D
D H
J
E G
F
F
G
I
G
G
I
J
H J
JI
1A D
4B C
4A B
Add Edge
29
4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
2
5
2
5
4
3
4
10
1
6
3
3
2
B
B
D
J
C
C
E
F
D
D H
J
E G
F
F
G
I
G
G
I
J
H J
JI
1A D
4B C
4A B
Cycle
Don’t Add Edge
30
4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
2
5
2
5
4
3
4
10
1
6
3
3
2
B
B
D
J
C
C
E
F
D
D H
J
E G
F
F
G
I
G
G
I
J
H J
JI
1A D
4B C
4A B
Add Edge
31
4
1
2
2 1
3
32
4
A
B C
D
E F
G
HI
J
4
1
2 3
2 1
3
5
3
4
2
5 6
4
4
10
A
B C
D
E F
G
HI
J
Minimum Spanning Tree Orginal Graph
32
Cost Computation• Cd(p) : matching cost for pixel p at disparity level d• : aggregated cost
-- σS and σR : constants used to adjust the similarity.
33
• Weight between p and q • w(p, q) = | I(p)-I(q)| = image gradient
• Distance between p and q • D(p, q) = sum of weights of the connected edges
• Similarity between p and q
• • Aggregated cost•
=>
Cost Aggregation on a Tree Structure
34
Cost Aggregation on a MST• Claim 1. Let Tr denote a subtree of a node s and r denote
the root node of Tr, then the supports node s received from this subtree is the summation of the supports node s received from r and S(s, r) times the supports node r received from its subtrees.• Supports r = • Supports s =
35
s
Tr
r
Cost Aggregation on a MST• Aggregated cost
=>
• , if node v is a leaf node• P(vc) denote parent of nodevc
36
Cost Aggregation on a MST
37
Cost Aggregation on a MST• Aggregated cost
=>
38
Cost Aggregation on a MST• Cost aggregation process• Aggregate the original matching cost Cd from leaf
nodes towards root node using Eqn. (6)• Aggregate from root node towards leaf nodes
using Eqn. (7)
• Complexity• Each level : 2 addition/subtraction + 3 multiplication
39
Disparity Refinement
•
• D : the left disparity map• Unstable : occlusion, lack of texture, specularity• Median filter overlap
40
Experimental Results_________________________
41
Experimental Results• Device : a MacBook Air laptop computer with a 1.8 GHz
Intel Core i7 CPU and 4 GB memory• Parameter :• σ = 0.1 (non-local cost aggregation)
• Source : Middlebury http://vision.middlebury.edu/stereo/ HHI database(book arrival) Microsofy i2i database(Ilkay)
42
Experimental Results• Time :• Proposed average runtime : 90 milliseconds (1.25× slower)• Unnormalized box filter average runtime : 72 milliseconds. • Local guided image filter average runtime : 960 milliseconds
43
[24] P. Viola and M. Jones. Robust real-time face detection.International Journal of Computer Vision, volume 57, pages 137–154, 2003.
[7] C.Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In CVPR ,2011.
44
[7] C.Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In CVPR ,2011.
Experimental Results
45
Experimental Results
46
Experimental Results
47
Experimental Results• Different disparity level (depth of spanning tree)
48
Max=7
49
Max=10
Max=14
50
Max=16
Max=20
51
Max=50
Max=75
Conclusion_________________________
52
Conclusion• Contributions• Outperform all local cost aggregation methods both in
speed and accuracy.• Present a near real-time stereo system with accurate
disparity results.• Future works• Apply to parallel algorithms• Refine matching cost estimation
53