From Pixels to Features:Review of Part 1

COMP 4900C

Winter 2008

Topics in part 1 – from pixels to features• Introduction

• what is computer vision? It’s applications.

• Linear Algebra• vector, matrix, points, linear transformation, eigenvalue,

eigenvector, least square methods, singular value decomposition.

• Image Formation• camera lens, pinhole camera, perspective projection.

• Camera Model• coordinate transformation, homogeneous coordinate,

intrinsic and extrinsic parameters, projection matrix.

• Image Processing• noise, convolution, filters (average, Gaussian, median).

• Image Features• image derivatives, edge, corner, line (Hough transform), ellipse.

General Methods

• Mathematical formulation• Camera model, noise model

• Treat images as functions

• Model intensity changes as derivatives• Approximate derivative with finite difference.

• First-order approximation

• Parameter fitting – solving an optimization problem

),( yxfI T

yx IIf ],[

fvujiIvIuIjiIvjuiI yx ),(),(),(

Vectors and Points

We use vectors to represent points in 2 or 3 dimensions

12

12

yy

xxPQvP(x1,y1)

Q(x2,y2)

x

yv

The distance between the two points:

212

212 )()( yyxxPQD

Homogeneous Coordinates

w

wy

wx

y

x

Go one dimensional higher:

w

wz

wy

wx

z

y

x

is an arbitrary non-zero scalar, usually we choose 1.w

32

31

3

2

1

/

/

xx

xx

x

x

x

43

42

41

4

3

2

1

/

/

/

xx

xx

xx

x

x

x

x

From homogeneous coordinates to Cartesian coordinates:

2D Transformation with Homogeneous Coordinates

2D coordinate transformation using homogeneous coordinates:

y

x

y

x

T

T

p

pp

cossin

sincos''

1100

cossin

sincos

1

''

''

y

x

y

x

y

x

p

p

T

T

p

p

2D coordinate transformation:

Eigenvalue and Eigenvector

We say that x is an eigenvector of a square matrix A if

xAx is called eigenvalue and is called eigenvector.

The transformation defined by A changes only themagnitude of the vector x

Example:

x

1

15

5

5

1

1

41

23

1

22

2

4

1

2

41

23and

5 and 2 are eigenvalues, and

1

1

1

2and are eigenvectors.

Symmetric Matrix

We say matrix A is symmetric if

AAT

Example: BBT

A symmetric matrix has to be a square matrix

is symmetric for any B, because

BBBBBB TTTTTT )()(

Properties of symmetric matrix:•has real eignvalues;•eigenvectors can be chosen to be orthonormal.• has positive eigenvalues.BBT

Orthogonal Matrix

A matrix A is orthogonal if

IAAT 1AATor

Example:

cossin

sincosA

cossin

sincos1A

The columns of A are orthogonal to each other.

Least Square

When m>n for an m-by-n matrix A, bAx has no solution.

In this case, we look for an approximate solution.We look for vector such that x

2bAx

is as small as possible.

This is the least square solution.

Least Square

Least square solution of linear system of equations

bAx

bAAxA TT

AAT is square and symmetric

bAAAx TT 1)(

Normal equation:

2bxA

The Least square solution

makes minimal.

SVD: Singular Value Decomposition

TUDVA An mn matrix A can be decomposed into:

U is mm, V is nn, both of them have orthogonal columns:

IUU T D is an mn diagonal matrix.

IVV T

Example:

10

01

00

30

02

100

010

001

00

30

02

Pinhole Camera

Why Lenses?

Gather more light from each scene point

Four Coordinate Frames

wim Pmatrix

tiontransformap

Camera model:

Zc

Xc

Yc

x

y

Xw

YwZw

Pw

xim

yim

cameraframe

imageplaneframe

worldframe

pixelframe

opticalcenter principal

point

imp

Perspective Projection

Z

Xfx

Z

Yfy

These are nonlinear.

10100

000

000

Z

Y

X

f

f

w

v

u

Using homogenous coordinate, we have a linear relation:

P

p

principalpoint

principalaxis

imageplane

opticalcenter

xy

wux / wvy /

World to Camera Coordinate

TRXX wc

1

1

1w

w

w

c

c

c

Z

Y

X

Z

Y

X

0

TR

Transformation between the camera and world coordinates:

R,T

Image Coordinates to Pixel Coordinates

ximx sxox )(

1100

/10

0/1

1

y

x

os

os

y

x

yy

xx

im

im

pixel sizes:, yx ss

x

y

(ox,oy)

xim

yim

yimy syoy )(

Put All Together – World to Pixel

1

100100

0010

0001

100

/0

0/

1

100100

000

000

100

/10

0/1

10100

000

000

100

/10

0/1

100

/10

0/1

3

2

1

w

w

w

yy

xx

w

w

w

yy

xx

c

c

c

yy

xx

yy

xx

Z

Y

X

TRosf

osf

Z

Y

X

TRf

f

os

os

Z

Y

X

f

f

os

os

w

v

u

os

os

x

x

x

1w

w

w

Z

Y

X

TRK

31 / xxxim 32 / xxyim

Camera Intrinsic Parameters

100

/0

0/

yy

xx

osf

osf

K

K is a 3x3 upper triangular matrix, called the Camera Calibration Matrix.

There are five intrinsic parameters:(a) The pixel sizes in x and y directions(b) The focal length(c) The principal point (ox,oy), which is the pointwhere the optic axis intersects the image plane.

yx ss ,f

Extrinsic Parameters

[R|T] defines the extrinsic parameters.The 3x4 matrix M = K[R|T] is called the projection matrix.

113

2

1

w

w

w

w

w

w

im Z

Y

X

MZ

Y

X

TRK

x

x

x

p

Image Noise

Additive and random noise:

I(x,y) : the true pixel values

n(x,y) : the (random) noise at pixel (x,y)

yxnyxIyxI ,,,ˆ

Gaussian Distribution

22 2/)(

2

1)(

xexp

Single variable

Gaussian Distribution

2

22

2 2exp

2

1,

yx

yxG

Bivariate with zero-means and variance2

Gaussian Noise

Is used to model additive random noise

2

2

2n

e

•The probability of n(x,y) is•Each has zero mean•The noise at each pixel is independent

Impulsive Noise

Salt-and-Pepper Noise:

• Is used to model impulsive noise

• Alters random pixels

• Makes their values very different from the true ones

x, y are uniformly distributed random variables

are constants

lxiiyi

lxkhIkhI sp

minmaxmin

,,

maxmin,, iil

Image Filtering

Modifying the pixels in an image based on some function of a local neighbourhood of the pixels

10 30 10

20 11 20

11 9 1

pN(p)

5.7

pf

Linear Filtering – convolution

1 3 0

2 10 2

4 1 1

Image

1 0 -1

1 0.1 -1

1 0 -1

Kernel

= 5

Filter Output

The output is the linear combination of the neighbourhood pixels

2/

2/

2/

2/

),(),(*),(m

mk

m

mhA kjhiIkhAAIjiI

The coefficients come from a constant matrix A, called kernel.This process, denoted by ‘*’, is called (discrete) convolution.

Smoothing by Averaging

1 1 1

1 1 1

1 1 19

1*

Convolution can be understood as weighted averaging.

Gaussian Filter

2

22

2 2exp

2

1,

yx

yxG

array mman ofelement an is , where khG

2

22

222

1),(

kh

ekhG

Discrete Gaussian kernel:

Gaussian Filter

*

1

Gaussian Kernel is Separable

2/

2/

22/

2/

2

2/

2/

2/

2/

2

2/

2/

2/

2/

),(

),(

),(),(

2

2

2

2

2

22

m

mk

km

mh

h

m

mh

m

mk

kh

m

mh

m

mk

G

kjhiIee

kjhiIe

kjhiIkhG

GII

2

2

2

2

2

22

222 khkh

eee

since

Gaussian Kernel is Separable

Convolving rows and then columns with a 1-D Gaussian kernel.

1 9 18 9 1

1

9

18

9

1

38

1=I Ir

Ir 38

1= result

The complexity increases linearly with instead of with .m 2m

Gaussian vs. Average

Gaussian Smoothing Smoothing by Averaging

Nonlinear Filtering – median filter

Replace each pixel value I(i, j) with the median of the valuesfound in a local neighbourhood of (i, j).

Median Filter

Salt-and-pepper noise After median filtering

Edges in Images

Definition of edges• Edges are significant local changes of intensity in an image.

• Edges typically occur on the boundary between two different regions in an image.

Images as Functions

2-D

Red channel intensity ),( yxfI

Finite Difference – 2D

h

yxfyhxf

x

yxfh

,,lim

,0

h

yxfhyxf

y

yxfh

,,lim

,0

Continuous function:

Discrete approximation:

jijix ff

x

yxfI ,,1

,

jijiy ff

y

yxfI ,1,

,

Convolution kernels:

11

1

1

Image Derivatives

11* II x

1

1*II y

Image I

Edge Detection using Derivatives

Pixels that pass thethreshold areedge pixels

1-D image )(xf

1st derivative )(' xf

threshold)(' xf

Image Gradient

yf

xf

f

gradient

22

yf

xff

)/arctan( xf

yf

magnitude

direction

Finite Difference for GradientDiscrete approximation:

jijix ffx

fjiI ,,1),(

jijiy ffy

fjiI ,1,),(

Convolution kernels:

11

1

1

magnitude ),(),(),( 22 jiIjiIjiG yx

direction )/arctan( xy II

aprox. magnitude yx IIjiG ),(

Edge Detection Using the Gradient

Properties of the gradient:

• The magnitude of gradient provides information about the strength of the edge

• The direction of gradient is always perpendicular to the direction of the edge

Main idea:

• Compute derivatives in x and y directions

• Find gradient magnitude

• Threshold gradient magnitude

Edge Detection Algorithm

11

1

1

*

*

xI

yI

22yx II

ThresholdImage I

edges

xI

yI

I

Edge Detection Example

I

),(),(),( 22 jiIjiIjiG yx

ThresholdjiG ),(

Edge Detection Example

Finite differences responding to noise

Increasing noise ->(this is zero mean additive gaussian noise)

Where is the edge?

Solution: smooth first

Look for peaks in

Sobel Edge Detector

101

202

101

121

000

121

Approximate derivatives with central difference

jijix ffx

fjiI ,1,1),(

101

Smoothing by adding 3 column neighbouring differences and givemore weight to the middle one

Convolution kernel for yI

Convolution kernel

Sobel Operator Example

101

202

101

)()(2)( 976431 aaaaaaI x

*

*

121

000

121

The approximate gradient at 5a

)()(2)( 938271 aaaaaaI y

987

654

321

aaa

aaa

aaa

987

654

321

aaa

aaa

aaa

Sobel Edge Detector

*

*

xI

yI

ThresholdImage I

101

202

101

121

000

121

edges22yx II

Edge Detection Summary

Input: an image I and a threshold .

1. Noise smoothing: (e.g. h is a Gaussian kernel)

2. Compute two gradient images and by convolvingwith gradient kernels (e.g. Sobel operator).

3. Estimate the gradient magnitude at each pixel

4. Mark as edges all pixels such that

hII s

),(),(),( 22 jiIjiIjiG yx

),( ji ),( jiG

xI yI sI

Corner Feature

Corners are image locations that have large intensity changesin more than one directions.

Shifting a window in any direction should give a large change in intensity

Harris Detector: Basic Idea

“flat” region:no change in all directions

“edge”:no change along the edge direction

“corner”:significant change in all directions

C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988

Change of Intensity

The intensity change along some direction can be quantifiedby sum-of-squared-difference (SSD).

ji

jiIvjuiIvuD,

2),(),(),(

v

u

),( jiI

),( vjuiI

Change Approximation

If u and v are small, by Taylor theorem:

vIuIjiIvjuiI yx ),(),(

wherey

IIand

x

II yx

therefore

v

u

III

IIIvu

vIuvIIuI

vIuI

jiIvIuIjiIjiIvjuiI

yyx

yxx

yyxx

yx

yx

2

2

2222

2

22

2

),(),(),(),(

Gradient Variation Matrix

v

u

III

IIIvuvuD

yyx

yxx2

2

),(

This is a function of ellipse.

2

2

yyx

yxx

III

IIIC

Matrix C characterizes how intensity changesin a certain direction.

Eigenvalue Analysis

QQIII

IIIC T

yyx

yxx

2

12

2

0

0

If either λ is close to 0, then this is not a corner, so look for locations where both are large.

(max)-1/2

(min)-1/2

yxy

xII

I

IC

AAIII

IC T

yxy

x

•C is symmetric•C has two positive eigenvalues

Corner Detection Algorithm

Line Detection

The problem:•How many lines?•Find the lines.

Equations for Lines

baxy

What happens when the line is vertical? The slope a goesto infinity.

The slope-intercept equation of line

A better representation – the polar representation

sincos yx

Hough Transform: line-parameter mapping

sincos yx

A line in the plane maps to a point in the - space.

(,)

All lines passing through a point map to a sinusoidal curve in the- (parameter) space.

Mapping of points on a line

Points on the same line define curves in the parameter spacethat pass through a single point.

Main idea: transform edge points in x-y plane to curves in theparameter space. Then find the points in the parameter spacethat has many curves passing through.

Quantize Parameter Space

m

m

Detecting Lines by finding maxima / clustering in parameter space.

Examples

Image credit: NASA Dryden Research Aircraft Photo Archive

input image Hough space lines detected

Algorithm

1. Quantize the parameter space int P[0, max][0, max]; // accumulators

2. For each edge point (x, y) { For ( = 0; <= max; = +) { // round off to integer (P[][])++; } }

3. Find the peaks in P[][].

sincos yx

Equations of Ellipse

12

2

2

21

2

r

y

r

x

022 feydxcybxyax

LetTyxyxyx ]1,,,,,[ 22x

Tfedcba ],,,,,[a

Then 0axT

Ellipse Fitting: Problem Statement

0),( axap Tf

fits best in the least square sense:ip

N

iiD

1

2),(min apa

),( apiDWhere is the distance from to the ellipse. ip

Given a set of N image points find the parameter vector such that the ellipse

Tiii yx ],[p

0a

Euclidean Distance Fit

iiiD ppap ˆ),(ip

ip̂

ip̂ is the point on the ellipse thatis nearest to ip

0),ˆ( apif

is normal to the ellipse at ii pp ˆ ip̂

Compute Distance Function

Computing the distance function is a constrained optimizationproblem:

2

ˆˆmin ii

i

ppp

subject to 0),ˆ( apif

Using Lagrange multiplier, define:

),ˆ(2ˆ),,(2

appp iii fyxL

Then the problem becomes: ),,(minˆ

yxLip

Set 0

y

L

x

L we have ),ˆ(ˆ appp iii f

where Ti yx ],[ˆ p

Ellipse Fitting with Euclidean Distance

Given a set of N image points find the parameter vector such that

Tiii yx ],[p

0a

N

i i

i

f

f

12

2

),(

),(min

ap

apa

This problem can be solved by using a numerical nonlinear optimization system.