Image Primitives and Correspondence

Stefano Soatto added with slides from Univ. of Maryland andR.Bajcsy, UCB

Computer Science DepartmentUniversity of California at Los Angeles

Siggraph 04 2

Given an image point in left image, what is the (corresponding) point in the rightimage, which is the projection of the same 3-D point

Image Primitives and Correspondence

Siggraph 04 3

Image primitives and Features

The desirable properties of features are: Invariance with respect to Grays

scale/color With respect to location (translation and

rotation) With respect to scale Robustness Easy to compute Local features vs. global features

Siggraph 04 4

Feature analysis

Points sensitive to illumination variation but fast to compute

Neighborhood features : gradient based (edge detectors) measuring contrast ,robust to illumination variation except for highlights

Fast computation ,it can be done in parallel.The complimentary feature to gradient is region

based. The advantage of this feature is it can encompass larger regions that are homogeneous and save processing time.

Siggraph 04 5

Profiles of image intensity edges

Siggraph 04 6

Image gradient The gradient of an image:

The gradient points in the direction of most rapid change in intensity

The gradient direction is given by:

how does this relate to the direction of the edge? The edge strength is given by the gradient

magnitude

Siggraph 04 7

The discrete gradient How can we differentiate a digital image f[x,y]?

Option 1: reconstruct a continuous image, then take gradient

Option 2: take discrete derivative (finite difference)

How would you implement this as a cross-correlation?

Siggraph 04 8

Effects of noise

Consider a single row or column of the image Plotting intensity as a function of position gives a signal

Where is the edge?

Siggraph 04 9 Where is the edge?

Solution: smooth first

Look for peaks in

Siggraph 04 10

2D edge detection filters

is the Laplacian operator:

Laplacian of Gaussian

Gaussian derivative of Gaussian

Siggraph 04 11

Effect of (Gaussian kernel size)

Canny with Canny with original

The choice of depends on desired behavior large detects large scale edges small detects fine features

Siggraph 04 12(Forsyth & Ponce)

Scale Smoothing Eliminates noise edges. Makes edges smoother. Removes fine detail.

Siggraph 04 13

Corners contain more edges than lines.

A point on a line is hard to match.

Corner detection

Siggraph 04 14

Corners contain more edges than lines.

A corner is easier

Siggraph 04 15

Edge Detectors Tend to Fail at Corners

Siggraph 04 16

Finding Corners

Intuition:

• Right at corner, gradient is ill defined.

• Near corner, gradient has two different values.

Siggraph 04 17

Formula for Finding Corners

2

2

yyx

yxx

III

IIIC

We look at matrix:

Sum over a small region, the hypothetical corner

Gradient with respect to x, times gradient with respect to y

Matrix is symmetric WHY THIS?

Siggraph 04 18

2

12

2

0

0

yyx

yxx

III

IIIC

First, consider case where:

What is region like if:

1. 1

2. 2

3. 1and 2

4. 1and 2

Siggraph 04 19

General Case:

From Linear Algebra, it follows that because C is symmetric:

RRC

2

11

0

0

With R a rotation matrix.

So every case is like one on last slide.

Siggraph 04 20

So, to detect corners

Filter image. Compute magnitude of the gradient

everywhere. We construct C in a window. Use Linear Algebra to find 1and 2. If they are both big, we have a corner.

Siggraph 04 21

Correspondence

Lambertian assumption

Rigid body motion

Matching - Correspondence

Siggraph 04 22

Translational model

Affine model

Transformation of the intensity values and occlusions

Local Deformation Models

Siggraph 04 23

Motion Field (MF)

The MF assigns a velocity vector to each pixel in the image.

These velocities are INDUCED by the RELATIVE MOTION btw the camera and the 3D scene

The MF can be thought as the projection of the 3D velocities on the image plane.

Siggraph 04 24

Motion Field and Optical Flow Field Motion field: projection of 3D motion vectors on image plane

Optical flow field: apparent motion of brightness patterns We equate motion field with optical flow field

00

00

10

0

00

ˆby torelated

imagein induces , velocity has point Object

zr

rrrr

rv

rv

vv

f

dt

d

dt

d

P

ii

i

i

Siggraph 04 25

2 Cases Where this Assumption Clearly is not Valid

(a) (b)

(a) A smooth sphere is rotating under constant illumination. Thus the optical flow field is zero, but the motion field is not.

(b) A fixed sphere is illuminated by a moving source—the shading of the image changes. Thus the motion field is zero, but the optical flow field is not.

Siggraph 04 26

Brightness Constancy Equation

Let P be a moving point in 3D: At time t, P has coords (X(t),Y(t),Z(t)) Let p=(x(t),y(t)) be the coords. of its

image at time t. Let E(x(t),y(t),t) be the brightness at p

at time t. Brightness Constancy Assumption:

As P moves over time, E(x(t),y(t),t) remains constant.

Siggraph 04 27

Brightness Constraint Equation

flow. optical of components the, ,, and irradiance thebe ,,Let yxvyxutyxE

expansionTaylor

,,,, tyxEtttvytuxE

0limit takingand by dividing

,,,,

tt

tyxEet

Et

y

Ey

x

ExtyxE

0

derivative total theofexpansion theiswhich

0

dt

dE

t

E

dt

dy

y

E

dt

dx

x

E

short: 0 tyx EvEuE

Siggraph 04 28

Brightness Constancy Equation

Taking derivative wrt time:Taking derivative wrt time:

Siggraph 04 29

Brightness Constancy Equation

LetLet(Frame spatial gradient)(Frame spatial gradient)

(optical flow)(optical flow)

andand (derivative across frames)(derivative across frames)

Siggraph 04 30

Brightness Constancy Equation

Becomes:Becomes:

vvxx

vv

yy

rr E E

The OF is CONSTRAINED to be on a line !The OF is CONSTRAINED to be on a line !

-E-Ett/|/|rr E| E|

Siggraph 04 31

Interpretation

Values of (u, v) satisfying the constraint equation lie on a straight line in velocity space. A local measurement only provides this constraint line (aperture problem).

Tyx

Tyx

tyx

n

EE

EE

EvuEE

,Let

,,

flow Normal

n

u

T

yx

ty

yx

txn

EE

EE

EE

EE

222 ,nnuu

Siggraph 04 32

• Normal flow

Aperture Problem

Siggraph 04 33

Recall the corner detector

The matrix for corner detection:The matrix for corner detection:

is singular (not invertible) when det(Ais singular (not invertible) when det(ATTA) A) = 0= 0

One e.v. = 0 -> no corner, just an edgeOne e.v. = 0 -> no corner, just an edgeTwo e.v. = 0 -> no corner, homogeneous regionTwo e.v. = 0 -> no corner, homogeneous region

Aperture Aperture Problem !Problem !

But det(ABut det(ATTA) = A) = ii = 0 -> one or both e.v. = 0 -> one or both e.v. are 0are 0

Siggraph 04 34

• Integrate over image patch

• Solve

Optical Flow

Siggraph 04 35

rank(G) = 0 blank wall problemrank(G) = 1 aperture problem rank(G) = 2 enough texture – good feature candidates

Conceptually:

In reality: choice of threshold is involved

Optical Flow, Feature Tracking

Siggraph 04 36

• Qualitative properties of the motion fields

• Previous method - assumption locally constant flow

• Alternative regularization techniques (locally smooth flow fields, integration along contours)

Optical Flow

Siggraph 04 37

Feature Tracking

Siggraph 04 38

3D Reconstruction - Preview

Siggraph 04 39

Harris Corner Detector - Example

Siggraph 04 40

Wide Baseline Matching

Siggraph 04 41

• Sum of squared differences

• Normalize cross-correlation

• Sum of absolute differences

Region based Similarity Metric

Siggraph 04 42

• Compute image derivatives • if gradient magnitude > and the value is a local maximum along gradient direction – pixel is an edge candidate

Canny edge detector

gradient magnitudeoriginal image

Edge Detection

Siggraph 04 43

x

y

• Edge detection, non-maximum suppression (traditionally Hough Transform – issues of resolution, threshold selection and search for peaks in Hough space)• Connected components on edge pixels with similar orientation - group pixels with common orientation

Non-max suppressed gradient magnitude

Line fitting

Siggraph 04 44

• Line fitting Lines determined from eigenvalues and eigenvectors of A• Candidate line segments - associated line quality

second moment matrixassociated with eachconnected component

Line Fitting

Siggraph 04 45

Take home messages

Correspondence is easy/difficult/impossible depending on the imaging constraints

Correspondence and reconstruction are tightly coupled problems, can be solved simultaneously [Jin et al., CVPR 2004]

For most scenes simple descriptors suffice to establish a few (50-500) corresponding points/lines

From now on just geometry