Epipolar lines

epipolar linesepipolar lines

BaselineO O’

epipolar plane

𝑝 ′𝑇 𝐸𝑝=0

Rectification

• Rectification: rotation and scaling of each camera’s coordinate frame to make the epipolar lines horizontal and equi-height,by bringing the two image planes to be parallel to the baseline

• Rectification is achieved by applying homography to each of the two images

Rectification

BaselineO O’

𝑞 ′𝑇𝐻 𝑙−𝑇 𝐸𝐻𝑟

−1𝑞=0

𝐻 𝑙 𝐻𝑟

Cyclopean coordinates• In a rectified stereo rig with baseline of length ,

we place the origin at the midpoint between the camera centers.

• a point is projected to:– Left image: , – Right image: ,

• Cyclopean coordinates:

Disparity

• Disparity is inverse proportional to depth• Constant disparity constant depth• Larger baseline, more stable reconstruction of depth

(but more occlusions, correspondence is harder)

(Note that disparity is defined in a rectified rig in a cyclopean coordinate frame)

Random dot stereogram

• Depth can be perceived from a random dot pair of images (Julesz)

• Stereo perception is based solely on local information (low level)

Moving random dots

Compared elements

• Pixel intensities• Pixel color• Small window (e.g. or ), often using

normalized correlation to offset gain• Features and edges (less common)• Mini segments

Dynamic programming

• Each pair of epipolar lines is compared independently

• Local cost, sum of unary term and binary term– Unary term: cost of a single match– Binary term: cost of change of disparity (occlusion)

• Analogous to string matching (‘diff’ in Unix)

String matching• Swing → String

S t r i n g

S w i n g

Start

End

String matching• Cost: #substitutions + #insertions + #deletions

S t r i n g

S w i n g

Dynamic Programming

• Shortest path in a grid• Diagonals: constant disparity• Moving along the diagonal – pay unary cost

(cost of pixel match)• Move sideways – pay binary cost, i.e. disparity

change (occlusion, right or left)• Cost prefers fronto-parallel planes. Penalty is

paid for tilted planes

Dynamic Programming

Start

, Complexity?

Probability interpretation: Viterbi algorithm

• Markov chain

• States: discrete set of disparity

• Log probabilities: product sum• Maximum likelihood: minimize sum of negative

logs• Viterbi algorithm: equivalent to shortest path

Markov Random Field

• Graph • In our case: graph is

a 4-connected grid

• Minimize energy of the form

• Interpreted as negative log probabilities

Iterated Conditional Modes (ICM)

• Initialize states (= disparities) for every pixel• Update repeatedly each pixel by the most likely

disparity given the values assigned to its neighbors:

• Markov blanket: the state of a pixel only depends on the states of its immediate neighbors

• Similar to Gauss-Seidel iterations• Slow convergence to bad local minimum

Graph cuts: expansion moves

• Assume is non-negative and is metric:

• We can apply more semi-global moves using minimal s-t cuts

• Converges faster to a better (local) minimum

α-Expansion

• Expansion move allows in one shot each pixel to either change its state to α or to maintain its current state

• We apply expansion moves for all possible disparity values and repeat this to convergence

• At convergence energy is within a scale factor from the global optimum:

where

α-Expansion (1D example)

α-Expansion (1D example)

𝛼

𝛼  

α-Expansion (1D example)

𝛼

𝛼  

α-Expansion (1D example)

𝐷𝑝(𝛼) 𝐷𝑞 (𝛼)

𝛼

𝛼  

𝑉 𝑝𝑞 (𝛼 ,𝛼 )=0

α-Expansion (1D example)

𝛼

𝛼  𝐷𝑝(𝑑𝑝) 𝐷𝑞 (𝑑𝑞)

But what about?

α-Expansion (1D example)

𝛼

𝛼  𝐷𝑝(𝑑𝑝) 𝐷𝑞 (𝑑𝑞)

𝑉 𝑝𝑞(𝑑𝑝 ,𝑑𝑞)

α-Expansion (1D example)

𝛼

𝛼  𝐷𝑝(𝑑𝑝)

𝑉 𝑝𝑞(𝑑𝑝 ,𝛼)𝐷𝑞 (𝛼)

α-Expansion (1D example)

𝛼

𝛼  𝐷𝑞 (𝑑𝑞)

𝑉 𝑝𝑞(𝛼 ,𝑑𝑞)𝐷𝑝(𝛼)

α-Expansion (1D example)

𝛼

𝛼  

𝑉 𝑝𝑞(𝛼 ,𝑑𝑞)𝑉 𝑝𝑞(𝑑𝑝 ,𝛼)

𝑉 𝑝𝑞(𝑑𝑝 ,𝑑𝑞)

Such a cut cannot be obtained due to triangle inequality:

Common Metrics

• Potts model:

• Truncated :

• Truncated squared difference is not a metric

Reconstruction with graph-cuts

Original Result Ground truth