Structure and Motion from Discrete Views

• Introduction

• Computing the fundamental matrix, F, from corner correspondences

• Feature matching

• RANSAC

• Estimation

• Determining ego-motion from F• Computing a homography, H, from corner correspondences

• SIFT for wide baseline matching

• More than two views

• Batch and sequential solutions

Cameras rotating about their centre

image plane 1

image plane 2

• The two image planes are related by a homography H

• H depends only on the relation between the image planes and camera centre, C, not on the 3D structure

P = K [ I | 0 ] P’ = K’ [ R | 0 ]

H = K’ R K^(-1)

Four points define a projective transformation

Automatic computation of a homography

• Given an image pair:

1. Compute interest points (feature detection)

2. Compute putative (hypothesised) correspondences

3. Compute the homography from the correspondences

Interest point detection

500 points in each image

Initial correspondences Final correspondences

Computing a panoramic mosaic

• camera rotates about its centre

• focal length can also change (zoom)

Choice of mosaic frame

• This produces the classic "bow-tie"mosaic.

Choose central image as reference

Wide Baseline Matching• Suppose we wish to compute F between any image pair (not restricted to small camera motion between views)

Wide Baseline Matching• Why does the short baseline computation fail?• Recall three steps:

1.1.Compute interest pointsCompute interest points2.2.Compute putative point correspondencesCompute putative point correspondences3.3.RANSAC estimation of F and consistent correspondence set.RANSAC estimation of F and consistent correspondence set.

Derivative of Gaussian

gdxdf ∗

f

gdxd

Source: S. Seitz

Edge

Derivativeof Gaussian

Edge = maximumof derivative

Second derivative of Gaussian

gdxdf 2

2

∗

f

gdxd

2

2

Edge

Second derivativeof Gaussian (Laplacian)

Edge = zero crossingof second derivative

Source: S. Seitz

Matched filter for blob detection

Scale-normalized Laplacian response

Unnormalized Laplacian responseOriginal signal

maximum

Blob detection in 2D• Laplacian of Gaussian: Circularly symmetric operator for

blob detection in 2D

⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

+∂∂

=∇ 2

2

2

222

norm yg

xgg σScale-normalized:

Scale-invariant feature detection

• Approximating the Laplacian with a difference of Gaussians:

( )2 ( , , ) ( , , )xx yyL G x y G x yσ σ σ= +

( , , ) ( , , )DoG G x y k G x yσ σ= −

(Laplacian)

(Difference of Gaussians)

Efficient implementation

Select canonical orientation

• Create histogram of local gradient directions computed at selected scale

• Assign canonical orientation at peak of smoothed histogram

• Each key specifies stable 2D coordinates (x, y, scale, orientation)

0 2π

Example

Feature Descriptor

• distribution of the gradient over an image patchgradient 3D histogram

→ →

image patch

y

x

4x4 location grid and 8 orientations (128 dimensions)

Epipolar geometry example

512 x 768 pixel images

interest points and scales

Now, match SIFT descriptors between images

All matches consistent with epipolar geometry

156 matches consistent with epipolar geometry

Epipolar geometry

Ex: Panoramas from an unordered image set

Autostitch, Mat Brown & David Lowe

Application: Post-production special effectsoriginal sequence

Augmentation