review of linear algebra - · pdf file3 2d-dsp • in summary: – image is a n x m...

2D DSP

Nuno Vasconcelos

UCSD

2

Images

• the incident light is collected by an image sensor

– that transforms it into a 2D signal

q

q2

Po

Pi

V

E

3

2D-DSP

• in summary:

– image is a N x M array of pixels

– each pixel contains three colors

– overall, the image is a 2D discrete-space

signal

– each entry is a 3D vector

– for simplicity, we consider only single

channel images

– but everything extends to color in a straightforward manner

},...,0{

},...,0{),,,(],[

2

121

Mn

Nnbgrnnx

},...,0{

},...,0{],,[

2

121

Mn

Nnnnx

4

Separable sequences

• a trivial concept,

– but probably the only real novelty in this lecture

– very important in practice, because it reduces 2D problem to

collection on 1D problems

• Definition: a sequence is separable if and only if

where f[.] and g[.] are 1D functions

• note: there are many examples of separable sequences

• but most sequences are not separable

][][],[ 2121 ngnfnnx

5

Linear Shift Invariant (LSI) systems

• straightforward extension of LTI systems

• Definition: a system T that maps x[n1,n2] into y[n1,n2] is

LSI if and only if

– it is linear

– it is shift invariant

],[],[

],[],[

],[],[

212211

212211

212211

nnbynnay

nnxbTnnxaT

nnbxnnaxT

],[],[ 22112211 mnmnymnmnxT

6

2D convolution

• the operation

is the 2D convolution of x and h

– we will denote it by

• this is of great practical importance:

– for an LSI system the response to any input can be obtained by

the convolution with this impulse response

– the IR fully characterizes the system

– it is all that I need to measure

],[],[],[ 22112121

1 2

knknhkkxnnyk k

],[],[],[ 212121 nnhnnxnny

7

2D convolution

• has various properties of interest

• but these are the ones that you have already seen in 1D

(check handout)

• some of the more important:

– commutative:

– associative:

– distributive:

– convolution with impulse:

xyyx

zyxzyx

zxyxzyx

],[],[],[ 2211221121 mnmnxmnmnnnx

8

2D convolution

• as in 1D, it is most easily done in graphical form

• e.g. how do we convolve these two sequences?

• we need four steps

n1

n2

n1

n2

(1) (2)*

(3) (4)

],[ 21 nnx ],[ 21 nnh

],[],[],[ 22112121

1 2

knknhkkxnnyk k

9

2D convolution

• step 1): express sequences in terms of (k1, k2)

k1

k2

k1

k2

(1) (2)

(3) (4)

],[ 21 kkx ],[ 21 kkh

],[],[],[ 22112121

1 2

knknhkkxnnyk k

10

2D convolution

• step 2): invert h(k1, k2)

k1

k2

(1)(2)

(3)(4)

],[],[ 2121 kkhkkg

],[],[],[ 22112121

1 2

knknhkkxnnyk k

k1

k2

(1) (2)

(3) (4)

],[ 21 kkh

k1

k2

(1) (2)

(3) (4)

],[ 21 kkh

1

2

11

2D convolution

• step 3): shift g(k1, k2) by (n1, n2)

k1

k2

(1)(2)

(3)(4)

],[],[ 2121 kkhkkg

],[],[],[ 22112121

1 2

knknhkkxnnyk k

k1

k2

(1)(2)

(3)(4)

],[

],[

2211

2211

knknh

nknkg

n1

n2

this sends

whatever is

at (0,0) to (n1,n2)

12

2D convolution

• step 4): point-wise multiply the two signals and sum

– e.g. for (n1,n2) = (1,0)

],[],[],[ 22112121

1 2

knknhkkxnnyk k

k1

k2

(1)(2)

(3)(4)

],[ 2211 knknh

n1

n2

k1

k2

],[ 21 kkx

x =k1

k2

],[ 21 nny

n1

n2

13

2D convolution

• step 4): point-wise multiply the two signals and sum

– e.g. for (n1,n2) = (2,0)

],[],[],[ 22112121

1 2

knknhkkxnnyk k

k1

k2

(1)(2)

(3)(4)

],[ 2211 knknh

n1

n2

k1

k2

],[ 21 kkx

x =k1

k2

],[ 21 nny

n1

n2

(3)

etc.

14

2D convolution

• is this the only way to look at convolution?

– any signal can be written as

– e.g.

],[],[],[ 22112121

1 2

knknkkxnnxk k

n1

n2

(a) (b)

n1

n2

(a)

n1

n2

(b)

= +

],1[],[ 2121 nnbnna

]0,0[x ]0,1[x

15

2D convolution

• we combine this

with the properties of the convolution

– commutative:

– associative:

– distributive:

– convolution with impulse:

• to obtain another interpretation

xyyx

zyxzyx

zxyxzyx

],[],[],[ 2211221121 mnmnxmnmnnnx

],[],[],[ 22112121

1 2

knknkkxnnxk k

16

2D convolution

• it is done like this

• note that this is just our definition of convolution

(no surprises here)

• but we want to think about it like this, not like this

],[*],[],[],[

],[*],[],[

],[*],[],[

2122112121

21221121

212121

1 2

1 2

nnhknknkkxnny

nnhknknkkx

nnhnnxnny

k k

k k

],[],[],[ 22112121

1 2

knknkkxnnxk k

17

2D convolution

• we can think of

as the following sequence of operations

1. set y[n1, n2] = 0, for all (n1,n2)

2. for each (k1,k2) such that x[k1,k2] is not zero

• set a = x[k1,k2]

• shift h[n1,n2] by (k1,k2)

• multiply the entire sequence by a

• add the entire sequence to y[n1,n2]

],[*],[],[],[ 2122112121

1 2

nnhknknkkxnnyk k

],[*],[],[ 21221121 nnhknknnnz a

],[],[],[ 212121 nnznnynny

18

2D convolution

• example

• (k1,k2) = (0,0)

n1

n2

n1

n2

(1) (2)*

(3) (4)

],[ 21 kkx ],[ 21 nnh

n1

n2

],[ 21 nny

=n1

n2

(1) (2)

(3) (4)

],[ 2211 knknh

+ 1xn1

n2

(1) (2)

(3) (4)

],[ 21 nny

19

2D convolution

• example

• (k1,k2) = (1,0)

n1

n2

n1

n2

(1) (2)*

(3) (4)

],[ 21 kkx ],[ 21 nnh

=n1

n2

],[ 2211 knknh

+ 1xn1

n2

(1) (3)

(3) (7)

],[ 21 nny

n1

n2

(1) (2)

(3) (4)

],[ 21 nny

(1) (2)

(3) (4)

(2)

(4)

20

2D convolution

• example

• (k1,k2) = (0,1)

n1

n2

n1

n2

(1) (2)*

(3) (4)

],[ 21 kkx ],[ 21 nnh

=n1

n2

(1) (3)

(4) (9)

],[ 21 nny

(2)

(4)

n1

n2

(1) (3)

(3) (7)

],[ 21 nny

(2)

(4)

n1

n2

],[ 2211 knknh

+ 1x(1) (2)

(3) (4)(3) (4)

21

2D convolution

• example

• (k1,k2) = (1,1)

n1

n2

n1

n2

(1) (2)*

(3) (4)

],[ 21 kkx ],[ 21 nnh

=n1

n2

(1) (3)

(4) (10)

],[ 21 nny

(2)

(6)

n1

n2

],[ 2211 knknh

+ 1x

(3) (7)

n1

n2

(1) (3)

(4) (9)

],[ 21 nny

(2)

(4)

(3) (4)

(1) (2)

(3) (4)(4)

22

2D convolution

• note the differences with the previous convolution

– before we were computing one y[n1,n2] at a time

– now we update the entire sequence at a time

k1

k2

(1)(2)

(3)(4)

],[ 2211 knknh

n1

n2

k1

k2

],[ 21 kkx

x =k1

k2

],[ 21 nny

n1

n2

(3)

=n1

n2

(1) (3)

(4) (10)

],[ 21 nny

(2)

(6)

n1

n2

],[ 2211 knknh

+ 1x

(3) (7)

n1

n2

(1) (3)

(4) (9)

],[ 21 nny

(2)

(4)

(3) (4)

(1) (2)

(3) (4)(4)

• When do I use the serial vs parallel method?

– serial always works

– parallel is useful when one of the sequences is small

– example

Convolution

23

n0 1 2 3 4n

-1

1

h[n] x[n]

• serial

Convolution

24

kn-1

n+1

k0 1 2 3 4

gn[n]=h[0]

gn[n+1]=h[-1]

gn[n-1]=h[1]

x[k]

n

k0 1 2 3 4

kn-1

n+1

n

n

0

1 2 3 4

-1

5

gn[k]=h[n-k]

n

-1

1

h[n]

x[n]*h[n]

serial

convolution

• parallel

Convolution

25

n0 1 2 3 4n

-1

1

h[n] x[n]

*

n0 1 2 3 4n

-1

1

=

*

n0 1 2 3 4n

-1

1*

+

• parallel

Convolution

26

n0 1 2 3 4n

-1

1

h[n] x[n]

*

n0 1 2 3 4

=

n

0 1 2 3

4

+n

0

1 2 3 4=

27

Separable systems

• Definition: a system is separable if and only if its

impulse response is a separable sequence

• note that, in this case the convolution simplifies

][][],[ 221121 nhnhnnh

1

1 2

1 2

1 2

],[][

][],[][

][][],[

],[],[],[

21111

22221111

22211121

22112121

k

k k

k k

k k

nkfknh

knhkkxknh

knhknhkkx

knknhkkxnny

28

Separable systems

• the convolution simplifies to

with

• note that:

– for a fixed k1, f[k1,n2] is 1D convolution of x[k1,n2] and h2[n2]

– for a fixed n2, y[n1,n2] is 1D convolution of f[n1,n2] and h1[n1]

1

],[][],[ 2111121

k

nkfknhnny

2

][],[],[ 2222121

k

knhkkxnkf

][],[],[ 222121 nhnkxnkf

][],[],[ 112121 nhnnfnny

29

Separable systems

• the convolution simplifies to a sequence of 1D steps

• step1) for every k1,

– f[k1,n2] is 1D convolution of x[k1,n2] and h2[n2]

– which means: “convolve the columns of x with h2 to obtain

columns of f”

][],[],[ 222121 nhnkxnkf

n1

n2

n1

n2

*

h2

],[ 21 nnx ],[ 21 nnf

30

Separable systems

• step2) for every n2,

– y[n1,n2] is 1D convolution of f[n1,n2] and h1[n1]

– which means: “convolve the rows of f with h1 to obtain rows of y”

n1

n2

n1

n2

*

h1

],[ 21 nny],[ 21 nnf

][],[],[ 112121 nhnnfnny

31

Separable systems

• in summary, if we have a separable system

to convolve with x[n1,n2] we:

– 1) “convolve the columns of x with h2 to create f”

– 2) “convolve the rows of f with h1 to obtain y”

][],[],[ 112121 nhnnfnny

][][],[ 221121 nhnhnnh

][],[],[ 222121 nhnkxnkf

32

The Discrete-Space Fourier Transform

• as in 1D, an important concept in linear system analysis

is that of the Fourier transform

• the Discrete-Space Fourier Transform is the 2D

extension of the Discrete-Time Fourier Transform

• note that this is a continuous function of frequency

• the nomenclature distinguishes it from the 2D Discrete

Fourier transform (we will get back to this)

• what does the DSFT of an image look like?

2121221

2121

2211

2211

1 2

),()2(

1],[

],[),(

ddeeXnnx

eennxX

njnj

njnj

n n

33

Image spectrum

• two images, the magnitude, and phase of their FTs

34

Phase and Magnitude

• curious fact

– all natural images have about the same magnitude transform

– monotonically decaying with frequency

• hence, phase seems to matter, but magnitude largely

doesn’t

• we can see this through the following experiment

• take two pictures, swap the phase transforms, compute

the inverse

• here is what you get

2

2

2

1

21

1),(

X

35

The importance of phase

Reconstruction with zebra

phase, cheetah magnitude

Reconstruction with cheetah

phase, zebra magnitude

36

LSI systems

• why do we care so much about Fourier transforms?

• note that when we convolve a sequence with a complex

exponential,

we get

),(],[

),(e e

e e ],[

],[ ],[],[

2121

21

jj

)(j)(j

21

22112121

2211

222111

1 2

1 2

Hnnx

H

kkh

knknxkkhnny

nn

knkn

k k

k k

2211],[ 21

njnjeennx

37

LSI systems

• but we have seen that, for an LSI system

– the output in response to x[n1,n2]

– is the convolution with the impulse response h[n1,n2]

– hence, the response to

– is

– this means that complex exponentials are the eigenfunctions of

LSI systems

– when we input an eigenfunction, we get back the same function

– but scaled by H(1,2)

– this is called the frequency response of the system

2211],[ 21

njnjeennx

),(],[ ],[ 212121 Hnnxnny

38

LSI systems

• this is remarkable, since

– we know that any signal can be represented as a weighted sum

of complex exponentials

– when the signal is fed to an LSI system, each exponential is

scaled by H(1,2)

– hence, the frequency response completely characterizes the

system

– and the DSFT of the output is just the product of the two

2121221

2121

2211

2211

1 2

),()2(

1],[

],[),(

ddeeXnnx

eennxX

njnj

njnj

n n

),(),(),( 212121 XHY

39

Example

• the system with impulse response on the left

• has frequency response

• note that:

– the response is 1 at DC

– lower for high frequencies

– this system is a low-pass filter

n1

n2

6

1

6

1

6

1

6

1

3

1

21

2121

cos3

1cos

3

1

3

1

6

1

6

1

6

1

6

1

3

1

],[),(

2121

2211

1 2

jjjj

njnj

n n

eeee

eennhH

40

WARNING

• WARNING, WARNING, WARNING!

• the equivalence

• is the oldest trick in the DSP book!

• please do not fall for it

– you can “read” the sequence that has this DSFT

by applying this trick and the definition of DSFT!

1cos2

11

nee

jnjn

2121 cos3

1cos

3

1

3

1 ),( H

n1

n2

6

1

6

1

6

1

6

1

3

1

41

WARNING

• Quizz: which on the left is the DSFT of

this image?

?

42

WARNING

• the way to think about this is the

following:

– this image has low frequencies

horizontally (1)

– high frequencies vertically (2)

– the spectrum is a delta function along

(1) and harmonics along (2)

– the spectrum is this

– wrong way: “because image is horizontal spectrum must be too”

43

Properties of the DSFT

• these are extremely important, but straightforward

extension of what you have seen in 1D

• only novelty is separability (homework):

– the DSFT of a separable sequence is itself separable

– it is the product of the DTFTs of the 1D sequences that make

up the 2D sequence

• all other properties carry from 1D to 2D

)()(),(][][],[ 221121221121 XXXnxnxnnx

44


45


46


47

Example

• consider the separable impulse response

• frequency response

• note that:

– this system is a high-pass filter

– “diagonal” frequencies are enhanced

)cos23)(cos23(

)()(),(

21

221121

HHH

n1

n2

(9) (-3)(-3)

(-3)

(-3)(3)

(-1)(-1)

h1(n1)

(3)

(-1)(-1)

h2(n2)

48

Examples

• what do filtered images look like?

– here is a noisy image

– a light square against dark background, plus noise

49

Examples

• what do filtered images look like?

– here is the magnitude of its DSFT (origin at center), it contains:

– a peak at the center,

– some background signal at all frequencies,

– a cross-like pattern that goes from low to high frequencies

– why does it look like this?

50

Examples

• one way to find out is to filter and reconstruct the image

– we simulate the ideal low-pass filter by

– removing all signal components outside a circle in the frequency

domain

– this is what the spectrum looks like

– this gets rid of the background signal that covers all frequencies

51

Examples• this is the resulting image

– the component we removed was due to the noise

– “white” noise has energy at all frequencies

– notice that there are some artifacts (i.e. ringing) in the

reconstructed image

52

Examples• what about the stuff other than noise?

– let’s high-pass by removing everything inside the circle

53


– we now get mostly noise, as expected

– note that the square has mostly gone away

– this means that the flat part is low-frequency

– but we can still see the edges

54

Examples• this is interesting

– the edges are not only low-pass

– maybe they are the reason for the cross-shaped pattern

– to check we band-pass

55


– we now get mostly the edges

– we were right, the edges cause the cross-shaped pattern

– note that the edges are very hard to filter out

56

Examples• this is one of the fundamental properties of images:

– edges have energy at all frequencies

original low-pass

band-pass high-pass

57

The z transform

• once again, it is a straightforward extension of 1D

• Definition: the z-transform of the sequence x[n1,n2] is

• the region of the (z1,z2) plane where this sum is finite is

called the Region of Convergence (ROC)

• it turns out that:

– in 2D the ROC is much more complicated than in 1D

– while in 1D the ROC is bounded by poles (0D subspace of the

2D complex plane)

– in 2D is bounded by pole surfaces (2D subspaces of the 4D

space of two complex variables)

21

1 2

212121 ],[),(nn

n n

zznnxzzX

58

The z-transform

• computation is also much harder:

– as you might remember from 1D

– most useful tool in computing z-transforms is polynomial

factorization

– z-transform is a ratio of two polynomials

– we factor in to a sum of low order terms, e.g.

– and then invert each of the terms to get y[n]

)(

)()(

zD

zNzY

i i zazY

11

1)(

59

z-transform

• in 2D we only have one of two situations

• 1) the sequence is separable, in which case everything

reduces to the 1D case

the proof is identical to that of the DSFT

• 2) the signal is not separable

– here our polynomials are of the form z1mz2

n and, in general, it is

not know how to factor them

– we can solve only if sequence is simple enough that we can do it

by inspection (from the definition of the z-transform)

)( of

and )( of :

)()(),(][][],[

222

111

221121221121

zXROCz

zXROCzROC

zXzXzzXnxnxnnx

60

Example

• consider the sequence

• the z-transform is

],[],[ 212121 nnubannx

nn

bzazbzaz

azaz

azazzzX

n n

nn

n n

nn

211

2

1

1

0 0

1

2

1

1

0 0

1

2

1

121

, ,1

1

1

1

),(

1 2

21

1 2

21

ROC

|z1|

|z2|

a

b

review of linear algebra - · pdf file3 2d-dsp • in summary: – image is a n x m...

Documents