Geometry Reconstruction

March 22, 2007

Fundamental Matrix

An important problem: Determine the epipolar geometry. That is, the correspondence between a point on one camera and its epipolar line on the other camera.

p

O

L

OR

Epipoles

Epipolar line

T

Geometry Reconstruction

Use eight-point algorithm, we can recover the fundamental matrix F.

Knowing the fundamental matrix is lot easier.

Geometry Reconstruction

Knowing the fundamental matrix, and a pair of corresponding pixels, we would like to obtain the 3D position of the corresponding scene point.

There are three cases:

1. Calibrated cameras and extrinsic parameters are known.

2. Calibrated cameras with unknown extrinsic parameters

3. Uncalibrated cameras.

Calibrated Cameras (Triangulation)

The geometric reconstruction is absolute (without ambiguity).

Pl

Pr

Calibrated Camera with Unknown Extrinsic Parameters

The geometric reconstruction is only up to a scale.

Pl

Pr

Main point: we don’t know T (the baseline of the system) and we have no way to ascertain the scale of the scene.

We have only the essential matrix or fundamental matrix to work with.

Calibrated Camera with Unknown Extrinsic Parameters

pr, pl are the left and right image points in camera coordinates

Intrinsic parameters allow to go from pixel coordinates to camera coordinates.

We get E from a few correspondences. But E is only determined up to a scale!

Calibrated Camera with Unknown Extrinsic Parameters

Pl

Pr

We have no T, no information on scale.

From E:

Et E = St S=

Find a set of (T, R).

Uncalibrated Cameras

We have two images, and that’s it!

The reconstruction is only up to a global projective transformation.

Uncalibrated Cameras

The ambiguity is easy to see.

Only F and pr, pl are known and F is known only up to a scale. (xl, xr are 4-by-1 vectors in homogeneous coordinates).

H a nonsigular 4x4 matrix

Uncalibrated Cameras

Projective Transform:

Given a 3D point, x=(x1, x2, x3). In homogenous coordinates, it is x= (x1, x2, x3, 1). If Hx = (y1, y2, y3, y4), then the image of the 3D point x under the projective transform H is (y1/y4, y2/y4, y3/y4).

It is a 15-dimensional (non-linear) transformation group.

It is important that we know there is ambiguity in reconstruction, but it is only up to a 15-dimensional transformation group. Ambiguity is global not local.

You have a weird camera…. A better camera perhaps.

Impossible result

Uncalibrated Cameras

Normalize Prt to [ I3 | 0 ]. Find a Pl

t that satisfy the equations above.

Pl = [ S F | e’ ] for some skew-symmetric matrix S and e’ the left epipole will do

Let S = [ e’ ] x

More about the class

1. We will cover two (and half) more topics: Shape from shading (differential geometry), optical flows (motions) and perhaps recognition (Chapters 10-13 in Horns’ book )

2. Office hours: Normally 2-4 on Friday. But come by anytime you need to discuss issues/problems with me. (Send email to see if I am in office.)

3. Assignments: To be discussed.

4. Solutions: Will be available starting today. TA has a busy semester so far.

5. Problem 4 will be available shortly ( couldn’t make the Monday deadline).

Summary

Metric Reconstruction

More sophisticated method using other constraints will reduce the projective ambiguity down to a global unknown similarity transform. Assume

1. both cameras have the same intrinsic parameters

2. Sufficiently many orthogonal lines have been identified.

Covered in advanced vision class.

Reconstruction From N-Views

(Projective) Reconstruction from a possibly large set of images.

Problem:

• Set of 3D points, Xj

• Set of cameras Pi

• For each camera, image points xji (the input data)

• Find Pi, Xj, such that Pi Xj = xji

N views and M points:

Total number of parameters: 11N+3M.

Number of Equations: NM

With enough points and views, we have number of equations > total number of parameters. The problem is over-constrained. (What about N=2?)

Reconstruction From N-Views

Tomasi and kanade

Factorization

Factorization

m is the number views and n is the number of points.

SVD and factorization

Projective Factorization

Iterative Optimization

State of the Art

Input: A video sequence

Output: Camera Matrices and 3D locations of the points (up to a global similarity transform).

Stereo Correspondence Problem

Solving Stereo Correspondence Problem

1. Intensity Correlation

2. Edge Matching

Intensity Correlation

Intensity Correlation

Intensity Correlation

Edge Correlation

Why Stereo Correspondence Problem Hard?

Distorted Subwindows if disparity is not constant (complicates correlation)

Why Stereo Correspondence Problem Hard?

Why Stereo Correspondence Problem Hard?