qingxiong yang, student member, ieee, liang wang, student member, ieee,

Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation,and Occlusion HandlingQingxiong Yang, Student Member, IEEE, Liang Wang, Student Member, IEEE,Ruigang Yang, Member, IEEE, Henrik Stewe´ nius, Member, IEEE, and David Niste´ r, Member, IEEE

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 31, NO. 3, MARCH 2009

Outline• Introduction• System Overview•Methods• Initialization• Pixel Classification• Iterative Refinement• Fast-Converging Belief Propagation

• Depth Enhancement• Experiments• Conclusion

Introduction

Introduction• Stereo is one of the most extensively researched

topics in computer vision.

• Energy Minimization framework:• Graph Cut• Belief Propagation(BP)

Objective(Contribution)• To formulate stereo model with careful handling

of:• Disparity• Discontinuity• Occlusion

• Differs from the normal framework in the final stages of the algorithm

• Outperforms all other algorithms on the average

System Overview

•1) Initialization

•1) Initialization

•1) Initialization

•2) Pixel Classification

•3) Iterative Refinement

Initialization(Block 1)

Initialization• Input:• Left Image IL

• Right Image IR

• Output:• Initial Left Disparity Map DL

(0)

• Initial Right Disparity Map DR

• Initial Data Term ED(0)

Image

Color-Weighted Correlation

Correlation Volume

Data Term Initialization

Hierarchical BP

Disparity Map Initialization

ED(0)

CL CR

DL(0) DR

Image

Color-Weighted Correlation

Correlation Volume

Data Term Initialization

Hierarchical BP

Disparity Map Initialization

Initialization• Color-weighted Correlation

• To build the Correlation Volume

• Makes the match scores less sensitive to occlusion boundaries

• By using the fact that occlusion boundaries most often cause color discontinuities as well

ED(0)

DL(0) DR

CL CR

Correlation Volume• Color difference Δxy between pixel x and y

(in the same image)

Ic: Intensity of the color channel c

• The weight of pixel x in the support window of y:

Color Difference Spatial Difference

10 21

Correlation Volume• The Correlation Volume[27]:

• Wx : support window around x• d(yL, yR ) : pixel dissimilarity[1]

• xL , yL : pixels in left image IL

• xR , yR : corresponding pixels in right image IR

• dx : disparity value of pixel XL in IL

weight

Pixels in the window Dissimilarity[1]

dx = arg min CL,x (yL, yR)

weight

xR = xL – dx

yR = yL – dx

Correlation VolumeDisparity Map Bad Pixel

[27] K.-J. Yoon and I.-S. Kweon, “Adaptive Support-Weight Approach for Correspondence Search,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, pp. 650-656, 2006.

[1] S. Birchfield and C. Tomasi, “A Pixel Dissimilarity Measure That Is Insensitive to Image Sampling,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, pp. 401-406, 1998.

Image

Color-Weighted Correlation

Correlation Volume

Data Term Initialization

Hierarchical BP

Disparity Map Initialization

Initialization• Initial Data Term• Total energy 　 = Data Term + Smooth Term

• Computed from Correlation Volume

• Given an iteration index i = 0 here because it will be iteratively refined

ED(0)

DL(0) DR

CL CR

Initial Data Term• Initial Data Term:

• Ƞbp : twice the average of correlation volume to exclude the outliers

Correlation Volume0.2

X 2Average

Correlation Volume

Image

Color-Weighted Correlation

Correlation Volume

Data Term Initialization

Hierarchical BP

Disparity Map Initialization

Initialization• hierarchical Belief Propagation

• Employed with the data term and the reference image

• Resulting in the initial left and right disparity maps DL

(0) and DR

DL(0) DR

CL CR

ED(0)

Pixel Classification

(Block 2)

Pixel ClassificationInput

Output

Pixel Classification•Mutual Consistency Check• Requires that the disparity value from the left and

right disparity maps are consistent, i.e.,

• Not Pass : occluded pixel• Pass : unoccluded pixel 　　　　 => Correlation Confidence Measure

• Correlation Confidence• Based on how distinctive the highest peak in a

pixel's correlation profile is

• 　 : the cost for the best disparity value• 　 : the cost for the second best disparity value 　　　　

Pixel Classification

If > αs stableElse unstable

0.04

dx = arg min CL,x (yL, yR)

Iterative Refinement

(Block 3)

• Goal: to propagate information from the stable pixels to the unstable and the occluded pixels

Input

Iteration

• Color Segmentation• Color segments in IL are extrated by Mean Shift[6]

Iterative Refinement

[6] D. Comaniciu and P. Meer, “Mean Shift: A Robust Approach Toward Feature Space Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, pp. 603-619, 2002.

• Plane Fitting

• Using the disparity values for the stable pixels in each color segment

• Disparity values are taken from the current hypothesis for the left disparity map DL

(i). (Initial: DL(0))

• The plane-fitted depth map is used as a regularization for the new disparity estimation.

Iterative Refinement

• Plane Fitting• Using RANSAC[10]

• Iterates until the plane parameters converge

Iterative Refinement[10] M.A. Fischler and R.C. Bolles, “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,” Comm. ACM, vol. 24, pp. 381-395, 1981.

• Plane Fitting output : D(i)

• The ratio of stable pixels of each segment:• If Ratio > ȠS

• Stable pixels: D(i)

• Unstable, Occluded pixels: D(i)

• If Ratio ≤ ȠS

• All pixels : D(i)

Iterative Refinementpf

L

pf

pf

0.7

Iteration

• Absolute Difference:

• D(i+1) : New Disparity Map• D(i) 　 : Plane-fitted Disparity Map

• Data Term:

Iterative Refinement

L

2.0

pf

0.5

0.05

• The core energy minimization of our algorithm is carried out via the hierarchical BP algorithm.

Belief Propagation

Total Energy for Pixel X

Data Term Smooth Term

•Max-Product BP[25] :

• 　　　　 : Message vector passed from pixel X to one of its neighbors Y

Max-Product Belief Propagation

Data Term

Jump Cost

[25] Y. Weiss and W. Freeman, “On the Optimality of Solutions of the Max-Product Belief Propagation Algorithm in Arbitrary Graphs,” IEEE Trans. Information Theory, vol. 2, pp. 732-735, 2001.

Max-Product Belief Propagation

xY

• Jump Cost:

• dx : Disparity of pixel X

• d : Disparity of pixel Y (X’s neighbor)

• αbp : The number of disparity levels / 8

• ρs : 1 – (normalized average color difference)

• ρbp : The rate of increase in the cost

Max-Product Belief Propagation

Disparity Differenceof pixel X and its neigbor Y

1

• Total Energy for pixel X:

• Finally the label d that minimizes the total Energy for each pixel is selected.

Max-Product Belief Propagation

Smooth TemData Tem

• Standard loopy BP algorithm is too slow.

• Hierarchical BP[9] runs much faster while maintaining comparable accuracy.

• Works in a coarse-to-fine manner

Hierarchical Belief Propagation

[9] P.F. Felzenszwalb and D.P. Huttenlocher, “Efficient Belief Propagation for Early Vision,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 261-268, 2004.

Hierarchical Belief PropagationCoarser(Level 1)

Finer(Level 0)

• A large number of iterations is required to guarantee convergence in a standard BP algorithm.

• Fast-Converging BP effectively removes the redundant computation.

• Only updating the pixels that have not yet converged (value bigger than ȠZ )

Fast-Converging Belief Propagation

0.1

Fast-Converging Belief Propagation

Depth Enhancement

• To reduce the discontinuities caused by the quantization

• Sub-pixel Estimation algorithm is proposed.

• Cost Function:

Depth Enhancement

• The depth with the minimum of the cost function:

• d: the discrete depth with the minimal cost• d+: d+1• d- : d-1

• Replace each value with the average of those values that are within one disparity over a 9 x 9 window

Depth Enhancement

Experiments

ExperimentsParameter Settings Used Throughout:

ExperimentsParameter Settings Used Throughout:

ExperimentsResults on the Middlebury Data Sets with Error Threshold 1

nonocc : The subset of the nonoccluded pixelsdisc : The subset of the pixels near the occluded areas. all : The subset of the pixels being either nonoccluded or half-occluded

Error%

ExperimentsResults on the Middlebury Data Sets with Error Threshold 0.5

Color-Weighted Correlation Voume :

Initial Hierarchical BP:

Plane fitting:

Integer-Based Disparity Map:

Depth Enhancement:

Ground Trueh:

Conclusion

Conclusion• Propose a stereo model based on • energy minimization• color segmentation• plane fitting• repeated application of hierarchical BP• depth enhancement

• A fast converging BP approach is proposed.• Preserves the same accuracy as the standard BP• The runtime is sublinear to the number of iterations.

Conclusion• The algorithm is currently outperforming the

other algorithms on the Middlebury data sets on average.

• There’s space for Improvement:• Only refined the disparity map for the reference

image• [19] suggests that, by generating a good disparity

map for the right image, the occlusion constraints can be extracted more accurately.

J. Sun, Y. Li, S.B. Kang, and H.-Y. Shum, “Symmetric Stereo Matching for Occlusion Handling,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 399-406, 2005.