the plan for today camera matrix part a) notation, preprocessing, and basic concepts. part b) 4...
Post on 21-Dec-2015
215 views
TRANSCRIPT
![Page 1: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/1.jpg)
The plan for today
Camera matrix Part A) Notation, preprocessing, and basic
concepts. Part B) 4 Stereo Algorithms
Slides are courtesy of Prof. Ronen Basri
![Page 2: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/2.jpg)
Stereo Vision
Objective: 3D reconstruction Input: 2 (or more) images taken with calibrated
cameras Output: 3D structure of scene Steps:
Rectification Matching Depth estimation
![Page 3: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/3.jpg)
Rectification
Image Reprojection reproject image planes onto
common plane parallel to baseline Notice, only focal point of camera
really matters(Seitz)
![Page 4: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/4.jpg)
Rectification
Any stereo pair can be rectified by rotating and scaling the two image planes (=homography)
We will assume images have been rectified so Image planes of cameras are parallel. Focal points are at same height. Focal lengths same.
Then, epipolar lines fall along the horizontal scan lines of the images
![Page 5: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/5.jpg)
Cyclopean Coordinates
Origin at midpoint between camera centers Axes parallel to those of the two (rectified) cameras
( / 2),
( / 2),
( ) ( ),
2 2
l l
r r
l r
l r l r
l r l r l r
f X b fYx y
Z Zf X b fY
x yZ Z
fbx x
Zb x x b y y fb
X Y Zx x x x x x
![Page 6: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/6.jpg)
Disparity
The difference is called “disparity” d is inversely related to Z: greater sensitivity to
nearby points d is directly related to b: sensitivity to small
baseline
l r
fbZ
x x
l rd x x
![Page 7: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/7.jpg)
Main Step: Correspondence Search What to match?
Objects?
More identifiable, but difficult to compute Pixels?
Easier to handle, but maybe ambiguous Edges? Collections of pixels (regions)?
![Page 8: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/8.jpg)
Random Dot Stereogram
Using random dot pairs Julesz showed that recognition is not needed for stereo
![Page 9: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/9.jpg)
Random Dot in Motion
![Page 10: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/10.jpg)
Finding Matches
![Page 11: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/11.jpg)
SSD error
disparity
1D Search More efficient Fewer false matches
![Page 12: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/12.jpg)
Finding Matches
Under what conditions pixels can be matched? Ignoring specularities, we can assume that
matching pixels have the same brightness (constant brightness assumption)
Still, changes in gain and sensitivity may change the values of pixels
![Page 13: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/13.jpg)
Possible solutions
Use larger windows
Constraint the search
Other metric e.g., Normalized correlation
![Page 14: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/14.jpg)
Window Size
W = 3 W = 20
![Page 15: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/15.jpg)
Constraining the Search
Restrict search to epipolar lines (1D search)
Enforce ordering
Problem: not always true
Enforce smoothness
Problem: discontinuities at object boundaries
![Page 16: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/16.jpg)
Ordering
![Page 17: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/17.jpg)
Ordering
![Page 18: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/18.jpg)
Summary basic ideas
Restrict search to epipolar lines (1D search) Use larger elements (larger windows, edges,
regions)Problem: large elements may be distorted
Enforce orderingProblem: not always true
Other similarity measures (e.g., Normalized correlation)
Enforce smoothnessProblem: discontinuities at object boundaries
![Page 19: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/19.jpg)
Comparison of Stereo Algorithms
D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms," International Journal of Computer Vision, 47 (2002), pp. 7-42.
Ground truthScene
![Page 20: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/20.jpg)
Scharstein and Szeliski
![Page 21: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/21.jpg)
Results with window correlation
Window-based matching(best window size)
Ground truth
![Page 22: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/22.jpg)
Graph Cuts
Ground truthGraph cuts
![Page 23: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/23.jpg)
Correspondence as Optimization Most stereo algorithms attempt to minimize a
functional that usually consists of two terms:
where
- penalizes for quality of a match (unary)
- penalizes non smooth (or even non fronto-parallel) reconstructions (binary)
Many different optimization approaches were proposed
match data smoothnessE E E
smoothnessEdataE
![Page 24: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/24.jpg)
Part B) 4 Stereo Algorithms
1. Dynamic programming
2. Minimal cut/Max flow
3. Space carving
4. Graph cut optimization
![Page 25: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/25.jpg)
?
![Page 26: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/26.jpg)
1D Methods: Dynamic Programming Discretize the 3-D space Find the correct curve at every slice
(A slice = epipolar plane)
![Page 27: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/27.jpg)
Dynamic programming
Find correspondences of each epipolar
line separately
![Page 28: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/28.jpg)
Dynamic programming
![Page 29: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/29.jpg)
Dynamic programming
How do we find the best curve? Assign weight of all edges
insertion
matchdeletion
![Page 30: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/30.jpg)
Dynamic programming
How do we find the best curve? Assign weight of all edges Find shortest path
Dijkstra
insertion
matchdeletion
![Page 31: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/31.jpg)
Results
![Page 32: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/32.jpg)
Dynamic programming
Advantages Simple, efficient Globally optimal
Disadvantages Each slice computed independently
(smoothness is not enforced between slices) Problems due to discretization (tilted planes)
![Page 33: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/33.jpg)
Stereo Algorithms
Brief review of 4 algorithms:
1. Dynamic programming
2. Minimal cut/Max flow
3. Space carving
4. Graph cut optimization
![Page 34: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/34.jpg)
Min Cut/Max Flow
![Page 35: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/35.jpg)
Min Cut/Max Flow
![Page 36: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/36.jpg)
Min Cut/Max Flow
![Page 37: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/37.jpg)
Min Cut/Max Flow
![Page 38: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/38.jpg)
Min Cut/Max FlowMain idea: Lets solve all DP problem
together.
Objective: find the optimal cut using all the slices simultaneously.
![Page 39: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/39.jpg)
Min Cut/Max Flow
![Page 40: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/40.jpg)
Min Cut/Max Flow
Construct a graph: Every voxel (3-D point in space) is a node Every node is connected to its 6 neighbors
![Page 41: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/41.jpg)
Min Cut/Max Flow
Weights on the edges: Data cost: change in pixel value
data
data
Neighbor
In next slice/row
Neighbor
In next slice/row
Neighbor Neighbor
![Page 42: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/42.jpg)
Min Cut/Max Flow
Weights on the edges: Data cost: change in pixel value Smoothness cost: change in depth
smooth
smooth
smooth
smooth
![Page 43: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/43.jpg)
Min Cut/Max Flow
Weights on the edges: Data cost: change in pixel value Smoothness cost: change in depth
data
![Page 44: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/44.jpg)
Min Cut/Max Flow
Add source and sink Find min cut
Source
…
…
Sink
∞
∞
![Page 45: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/45.jpg)
Min Cut/Max Flow
Data penalty
Smoothness penalty
![Page 46: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/46.jpg)
Results
Input Min cut Dynamic programming
![Page 47: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/47.jpg)
Min Cut/Max Flow
Advantages All slices are optimized simultaneously Efficient
Disadvantages Extension to multi-camera is difficult Discretization
![Page 48: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/48.jpg)
Multi-view stereo Every point in space
corresponds to a match in the images
Compute data term for each match
Space Carving
![Page 49: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/49.jpg)
0.5 0.4 0.8 0.9 0.9 0.8 0.9 0.3 0.2
Space Carving
Multi-view stereo Every point in space
corresponds to a match in the images
Compute data term for each match (“photo-consistency”)
![Page 50: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/50.jpg)
Space Carving
Dynamic data term (taking occlusion into account)
Order of sweep is important
![Page 51: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/51.jpg)
Space Carving
![Page 52: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/52.jpg)
Space Carving
Done for all slices simultaneously
![Page 53: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/53.jpg)
Space Carving
Done for all slices simultaneously
![Page 54: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/54.jpg)
Space Carving
Done for all slices simultaneously
![Page 55: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/55.jpg)
Space Carving
Computes a bound on the object, the visual hull More camera views: better result
![Page 56: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/56.jpg)
Space Carving: Results
![Page 57: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/57.jpg)
Space Carving: Results
![Page 58: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/58.jpg)
Space Carving
Advantages True multi-views stereo Handles occlusion
Disadvantages Limited to visual hull Lacks smoothness term Noise may introduce holes,
allowing for noise may thicken shape
Discretization
![Page 59: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/59.jpg)
Graph Cut Optimization
Stereo is a minimization problem
Possible solution: local search (gradient descent) Problem: inefficient, local minima Instead, search larger areas at every iteration
match data smoothnessE E E
![Page 60: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/60.jpg)
Graph Cut Optimization
Construct a graph to represent the problem: Nodes:
Pixels (in first image) k discrete depth values
Edges: From every pixel node to a
depth node (data term) Neighboring nodes (smoothness)
Assign weights corresponding to pixel intensities to get a global cost function
pixels
depths
…21 k
![Page 61: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/61.jpg)
Graph Cut Optimization
Objective: Multiway cut Edges:
Every pixel remains connected to one depth node
Edges between neighboring nodes only if they are connected to same depth node
Nodes are assigned the depth that they are connected to
Multiway cut is NP-complete, solve iteratively
…
……21 k3
pixels
depths
![Page 62: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/62.jpg)
Graph Cut Optimization
α-β swap Nodes labeled α or β, (i.e.,
connected to or )
can change their labeling to α or β
Edges between neighbors are updated according to the new labeling
Other edges are not changed Finding best swap = min cut!
α β
…
……21 k3
pixels
depths
![Page 63: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/63.jpg)
Graph Cut Optimization
Example: 1-2 swap
…
…… k
…
…
… k1 2 1 23 3
![Page 64: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/64.jpg)
Graph Cut Optimization
Example: 1-2 swap
…
…
… k1 2 3 … k321
Connect the nodes labeled 1 or 2 to both
labels
![Page 65: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/65.jpg)
Graph Cut Optimization
Example: 1-2 swap… k3
2
1 … k3
2
1
Mark 1 as source and 2 as sink Find minimal cut
![Page 66: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/66.jpg)
Graph Cut Optimization
Example: 1-2 swap
… k3
2
1
… k1 2 3
Erase edges that were on the cut
Result: a new labeling of the 1,2 nodes
![Page 67: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/67.jpg)
Graph Cut Optimization
Start with an arbitrary labeling For every pair {α, β} є {1,…,k}
Find the best α-β swap (minimizing the function) Update the graph (add and erase edges)
Quit when no pair improves the cost function Induce pixel labels
![Page 68: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/68.jpg)
Graph Cuts: Results
![Page 69: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/69.jpg)
…21 k3
Advantages State of the art results Efficient Bound on approximation quality Same technique can be applied to other
problems (e.g., image restoration)Disadvantages Discretization Occlusion Still room for improvement
Graph Cut Optimization
![Page 70: The plan for today Camera matrix Part A) Notation, preprocessing, and basic concepts. Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d565503460f94a350f5/html5/thumbnails/70.jpg)
Summary
Stereo vision: shape reconstruction from two or more images
Steps: Rectification Correspondence search Depth estimation
Algorithms: Dynamic programming Min cut/max flow Space carving Graph cuts