introduction to fourier processing a spectrum of possibilities…

Introduction to Fourier Processing

A spectrum of possibilities…

04/19/23 Copyright, G. A. Tagliarini, PhD2

Organization

Signals: analog and discrete The Fourier Transform

– Definition– Discretization– Nyquist limits– Computation via the FFT

Applications– Correlation– Convolution– Filtering

Two-dimensional FFTs


A Signal

sum

-3

-2

-1

0

1

2

3

0 0.2 0.4 0.6 0.8 1 1.2

sum


Given:

An analog signal xa(t), t ≥ 0 The signal is sampled at uniform intervals

of length T to produce a discrete time signal x(k) ≡ xa(kT), k = 0, 1, 2,…

The signal x(k) is said to have a sampling frequency fs = 1/T

The sampled signal was periodic


The Fourier Transform (FT)

.1

where

:as defined is

, written , of ansformFourier tr the

,- , signal analogan Given

2

j

dtx(t) eX(f)

X(f)x(t)

tx(t)

πftj


The Inverse Fourier Transform (FT-1)

.1

again where

:by defined FT inverse theusing

ansformFourier tr its from

, signal at reconstruccan We

2

j

dfX(f)ex(t)

X(f)

x(t)

πftj


Observations

The Fourier transform maps real-valued functions of t (time) into complex-valued functions of f (frequency)

One can reconstruct a time signal by using FT-1

Since we will work with discrete samples from signals, our digital signal processing (DSP) will typically not require analytical solutions of integral equations!!!


The Energy Density Spectrum of x(t)

The energy density spectrum, a.k.a. power spectrum, of a signal x(t) is given by |X(f)|2

If x(t) is real, X(-f) = X*(f), the complex conjugate of X(f)

If 0≤f0≤f1, the energy in the band from f0 through f1 is given by:

1

0

210 |)(|2),(

f

fdffXffE


Example 1: Energy Density Spectrum

See example for FFT and PSD


Bandwidth Limited Signals

If a value B>0, such that X(f) = 0 for all values of f>B, then x(t) is said to be bandwidth limited

Bandwidth limited signals occur often– Bio-mechanical properties limit the highest frequency

signals that animals can generate naturally: Vocalizations in humans (40 Hz – 20 KHz) Vocalizations in dolphins and whales (up to 100 KHz)

– Electrical and physical characteristics of components also constrain frequency ranges in electrical circuits

Common analog telephone circuits (100 - 3,000 Hz) FM radio band width (15 - 30 KHz)


Nyquist Sampling

Recall that the sampling frequency, fs, is the reciprocal of the sampling interval T, i.e., fs = 1/T

Suppose a signal x(t) is bandwidth limited to B Hz By the sampling theorem, if fs ≥ 2 B, the signal can

be approximated (interpolated) between samples. The Nyquist critical frequency is fcN = 2 B

The critical sampling interval Tc = 1/(2B)


Nyquist Sampling: Interpretation

Given a signal x(t) with bandwidth limit B:– The Nyquist critical frequency is fcN = 2 B– The critical sampling interval Tc = 1/(2B)

Meaning– To reconstruct x(t) without distortion x(t), the sampling rate

must be at least twice the highest frequency present– The maximum sampling interval is one half the wavelength

of the highest frequency component The sampling process must obtain at least two

samples per cycle of the highest frequency present!


Interpolating Between Samples

Assume– x(t) is bandwidth limited by B– The sampling frequency fs ≥ 2B, where T = 1/ fs

Then, by the sampling theorem we can interpolate between samples using:

k s

s

kTtf

kTtfkTxtx

)(

))(sin()()(


One More Useful Notion

Recall a definition of the definite integral

n

abx

xxkafdxxfn

kn

b

a

Here exists.limit when the

)(lim)(1

0


The Discrete Fourier Transform

Given the definition of the Fourier Transform

The discrete Fourier transform (DFT) is given by:

dtx(t) eX(f)πftj2

Nkm(kT)xx(k)

ekxmX

a

N

k

N

mkj

,0 and where

,)()( 1

0

2


The Discrete Fourier Transform

The discrete Fourier transform (DFT):

approximates the Fourier transform at the N frequency values determined by m, 0≤m<N

1

0

2

)()(N

k

N

mkj

ekxmX


The Inverse of the Discrete Fourier Transform

Recall the inverse Fourier transform is:

The inverse of the DFT is given by:

1

0

2

)(1

)(N

m

N

mkj

emXN

kx

dfX(f)ex(t)πftj 2


Example 2: A Two-sample DFT

Suppose – N = 2– Samples x0, x1 are given corresponding to xa(kT),

for T = 0, 1

Find X(m)=?


Example 2: A Two-sample DFT (continued)

1,0for ,)(

from obtained is DFT the,& samplesGiven 1

0

2

2

10

mexmX

xx

k

mkj

k


Example 2: A Two-sample DFT (continued)

10

1

0

1

0

1

0

1

10

1

0

1

0

1

0

01

0

0

1

0

1

0

2

2

10

)0)(cos(

))sin()(cos()1(

)01(

))0sin()0(cos()0(

thatso 1,0for ,)(

from obtained is DFT the,& samplesGiven

xxjkx

kjkxexX

xxx

jxexexX

mexexmX

xx

kk

kk

k

kjk

kk

kk

kk

k

kjk

k

mkjk

k

mkj

k


Example 3: Four Sample DFT

Suppose – N = 4– Samples x0, x1, … x3 are given corresponding to

xa(kT), for k = 0,…,3

Find X(m)=?


Example 3: A Four Sample DFT (continued)

3,...,0for ,)(

from obtained is DFT the,,..., samplesGiven 3

0

23

0

4

2

30

mexexmX

xx

k

mkj

kk

mkj

k



3

03210

3

0

03

0

2

0

3

0

23

0

4

2

30

)0(

3,...,0for ,)(

from obtained is DFT the,,..., samplesGiven

kk

kk

k

kj

k

k

mkj

kk

mkj

k

xxxxxexexX

mexexmX

xx



?)1(

3,...,0for ,)(


3

0

23

0

2

1

3

0

23

0

4

2

30

k

kj

kk

kj

k

k

mkj

kk

mkj

k

exexX

mexexmX

xx



?)2(

3,...,0for ,)(


3

0

3

0

2

2

3

0

23

0

4

2

30

k

kjk

k

kj

k

k

mkj

kk

mkj

k

exexX

mexexmX

xx



?)3(

3,...,0for ,)(


3

0

2

33

0

2

3

3

0

23

0

4

2

30

k

jk

kk

kj

k

k

mkj

kk

mkj

k

exexX

mexexmX

xx


Example 4: An Eight Sample DFT

Suppose – N = 8– Samples x0, x1, … x7 are given corresponding to

xa(kT), for T = 0,…,7

Find X(m)=?


Example 4: An Eight Sample DFT (continued)

7,...,0for ,)(

from obtained is DFT the,,..., samplesGiven 7

0

47

0

8

2

70

mexexmX

xx

k

mkj

kk

mkj

k


Example 4: Fourier Transform Computations with m=0

?)01(

)0sin0(cos)0(

then 0, mLet

7

0

7

0

7

0

4

0

kk

kk

k

kj

k

jx

jxexX


Example 4: Fourier Transform Computations with m=0 (continued)

7

076543210

7

0

7

0

4

0

)0sin0(cos)0(

then 0, mLet

kk

kk

k

kj

k

xxxxxxxxx

jxexX


Example 4: Fourier Transform Computations (continued)

?)4

sin4

(cos)1(

then 1, mLet 7

0

7

0

4

1

kk

k

kj

k

kj

kxexX



)4

7sin

4

7(cos)

4

6sin

4

6(cos

)4

5sin

4

5(cos)

4

4sin

4

4(cos

)4

3sin

4

3(cos)

4

2sin

4

2(cos

)4

1sin

4

1(cos)

4

0sin

4

0(cos

)4

sin4

(cos)4

sin4

(cos)1(

76

54

32

10

7

0

7

0

jxjx

jxjx

jxjx

jxjx

kj

kx

kj

kxX

kk

kk



)2

2

2

2()10( )

2

2

2

2()01(

)2

2

2

2()10( )

2

2

2

2()01(

)4

sin4

(cos)4

sin4

(cos)1(

7654

3210

7

0

7

0

jxjxjxjx

jxjxjxjx

kj

kx

kj

kxX

kk

kk


Example 4: Fourier Transform Computations (continued)

?)2

sin2

(cos

)2

sin2

(cos)2(

then 2, mLet

7

0

7

0

7

0

4

2

kk

kk

k

kj

k

kj

kx

kj

kxexX



)10(x)01(x)10(x)01( x

)10()01()10()01(

)2

sin2

(cos

)2

sin2

(cos)2(

then 2, mLet

7654

3210

7

0

7

0

7

0

4

2

jjjj

jxjxjxjx

kj

kx

kj

kxexX

kk

kk

k

kj

k


Reducing the Computations

The definition of the DFT naturally gives rise to an O(n2) algorithm for implementation

There is considerable computational symmetry that arises from taking sines and cosines at regularly spaced binary subdivisions of the complex unit circle |z|=1


Useful Notation

m

mjmN

N

jmN

N

jmN

N

N

j

N

m

mjmeeeW

Nj

NeW

1)cos(

)sin()cos(

thatnote and

2sin

2cos Define

2

222

2

2


Writing the DFT With the WN Notation

?

)(

split becan which ,)(

have we,2N and Given

12

0

2

2

12

0

1

2

12

0

1

0

1

0

2

p2

N

n

mN

n

NNn

N

k

kmNk

N

Nk

kmNk

N

k

kmNk

N

k

kmNk

N

k

N

kmj

k

N

j

N

WxWx

WxWxmX

WxexmX

eW


Writing the DFT With the WN Notation (continued)

mN

n

nmNN

n

N

k

kmNk

N

n

Nm

Nnm

NNn

N

k

kmNk

N

n

mN

n

NNn

N

k

kmNk

WxWx

WWxWx

WxWxmX

1

)(

12

0 2

12

0

12

0

2

2

12

0

12

0

2

2

12

0


Writing the DFT With the WN Notation (continued)

12

0 2

12

0 2

12

0

12

0 2

12

0

1

1

1)(

N

k

kmNN

k

mk

N

n

nmNN

n

m

N

k

kmNk

m

N

n

nmNN

n

N

k

kmNk

Wxx

WxWx

WxWxmX


So What? Some Improvement

This sum has cut the number multiplications in half; i.e., a Fourier transform of length N=2p , which normally requires sums of N terms each including a product can be computed from a sum having only N/2 products

The number of sums has stayed the same Can we do better? What happens if we consider the even and odd

terms separately?


Considering Even and Odd Terms

12

0

12

0

2

2

2

2

21)2(

N

k

N

k

mkNN

kk

mkNN

k

mk WxxWxxmX

kN

N

k

mkNN

kk

kN

N

k

mkNN

k

mk

WWxx

WWxxmX

12

0

2

2

12

0

2

2

12

1)12(

12

0 2

1)(

N

k

kmNN

k

mk WxxmX



12

0

2

12

0

2

2

)2(

N

k

mkNk

N

k

mkNN

kk WaWxxmX

12

0

2

12

0

2

2

)12(

N

k

mkNk

kN

N

k

mkNN

kk WbWWxxmX

12

0 2

1)(

N

k

kmNN

k

mk WxxmX



DFT term2

Nan )2(

12

02

12

0

2

N

k

mkNk

N

k

mkNk WaWamX

DFT term-2

Nan )12(

12

02

12

0

2

N

k

mkNk

N

k

mkNk WbWbmX

12

0 2

1)(

N

k

kmNN

k

mk WxxmX


Observations

An N-point DFT can be computed by calculating two N/2-point DFTs, each requiring half the effort – DFT is O(N2) – The subdivided effort is 2 O((N/2)2) = O(N2/2)

This process can be iterated over log2(N) levels

Thus, the FFT is O(1/2 N log2(N))


An FFT Algorithm (Cooley and Tukey)

1. Set θ = -2πd/N and r = N/22. For i = 1 to N-1 do //Calculate FT

1. Set w = cos(iθ)+j sin(iθ)2. For k = 0 to N-1 do

1. Set u = 12. For m = 0 to r-1 do

1. t = z(k+m) - z(k+m+r) 2. z(k+m) = z(k+m) + z(k+m+r)3. z(k+m+r) = tu4. u = w u

3. k = k + 2r3. i = 2i and r = r/2


An FFT Algorithm (continued)

3. For i = 0 to N-1 do //Sort results1. Set r = i and k = 02. For m = 1 to N-1 do //Bit reversal

1. k = 2k + (r % 2)2. r = r/23. m = 2m

3. If k > i do //swap z(i) and z(k); note t is complex1. t = z(i)2. z(i) = z(k)3. z(k) = t


An FFT Algorithm (continued)

4. If d < 0 //Find FFT-1, Scale resultsFor i = 0 to N-1 do

z(i) = z(i)/N


Interpreting the Results

What are the frequencies fk?fk = (k fs) / N, 0 ≤ k < N/2

Recall X(-f) = X*(f) if x(t) is real Spectra

– Amplitude: A(m) ≡ |X(m)|– Phase: φ(m) ≡ arctan( Im(X(m)) / Re(X(m)) )– Power (energy): PSD(m) ≡ |X(m)|2


Correlation: Background

x(k) = u(k-d) + v(k)– x(k) is the received signal – u(k) is the transmitted signal attenuated by – v(k) is additive random noise


Correlation: Definition

Given a signal x(k) of length N and y(k) of length M≤N

Assume y is padded with zeros to length N The cross correlation rxy of x with y is given

by:

NiikykxirN

ikxy

1

0),()()(


Correlation:

NiikykxirN

ikxy

1

0),()()(

x(N-1)…x(0) … x(i)

y(0)

y(N-1)…y(0) … …

y(N-1)…y(0) … y(N-1-i)

i=0

i=N-1… y(N-1)

i


Correlation: Fast Computation

The cross correlation can be computed quickly using the FFT and FFT-1

rxy (k) = FFT-1 (X(m)Y*(m)), 0≤m<N Here

– X(m) = FFT(x(k)) and – Y*(m) = FFT(y(k))


Convolution: Introduction

A dynamic system is one which exhibits memory analogously to a sequential circuit

Let – u(k) represent the system input at time k– y(k) represent the system output at time k

We write y(k)=f(y(k-1), y(k-2),…, y(k-n), u(k), u(k-1), u(k-2),…,u(k-m)), i.e., the system output depends on some history as well as the current input


Convolution: Linear Dynamic System

0 ),(

bygiven is

system timediscretelinear then,combinatiolinear a is f If

01

kjkubi)y(kay(k)m

jj

n

ii


Convolution: Linear Dynamic System

If ai = 0 for i = 1,…, n, the system forms a moving average (MA) model

If bj = 0 for j = 1,…, m, the system forms an auto-regressive (AR) model

If there are both i and j for which ai ≠ 0 and bj ≠ 0, the system forms an auto-regressive, moving average (ARMA) model


Convolution: Pulse Response 1

A causal signal is one for which u(k)=0 for k<0 Suppose also that y(k)=0 for k<0 (initial conditions

are zero) Let (k)≡1 if k=0 and 0 if k≠0 The output of a linear system H, generated by the

unit pulse input (k) with initial conditions of zero is called the pulse response of the system, denoted by h(k) ≡ H((k)) (Equation *)


Convolution: Pulse Response 2

Compute the pulse response h(k) of a system by letting u(k) = (k)

Also note that

where u(i) scales shifted pulses (k-i)

0,)()()(0

kikiukuk

i


Convolution:

By the linearity in Equation *, h(k) = H((k)), we have– H(u(i) (k-i)) = u(i) h(k-i), so that

This is the convolution of h(k) with u(k)

0,)()()(0

kiuikhkyk

i


Fast Convolution

Given input u(k) and pulse response h(k)– U(m) = FFT(u(k))– H(m) = FFT(h(k))

The Fast Convolution y(k) is given by y(k) = FFT-1(H(m) U(m))


Example: Averaging Filter

Let – h(k) = 1/p, for 0 ≤ k < p and – h(k) = 0 elsewhere

The convolution of h(k) with an input u(k) implements a MA filter of length p

This filter calculates an average of p samples of the input signal u


Digital Filtering: Notation (Again!!)

Let– Input signal u(k) with FT(u(k)) = U(m)– Pulse response h(k) with FT(h(k)) = H(m)– Output signal y(k) with FT(y(k)) = Y(m)– 0 ≤ m < N

Y(m) = H(m) U(m)


Digital Filtering: Polar Notation 1

The polar notation for the Fourier transforms U(m) = Au(m) exp(j u(m))

Y(m) = Ay(m) exp(j y(m)) H(m) = A(m) exp(j (m))

Where A(m) and (m) are the magnitude and phase response of the filter respectively



Since

Ay(m) exp(j y(m)) = Y(m) = H(m) U(m)

= A(m) exp(j (m)) Au(m) exp(j u(m))

= A(m) Au(m) exp(j (m)) exp(j u(m))

= A(m) Au(m) exp(j ((m)+u(m)))

Output Amplitude is the product of filter and signal amplitude Phase is sum of the filter and signal phases



Again Output amplitude is the product of filter and

signal amplitude spectra, Ay(m) = A(m) Au(m)

Output phase is sum of the filter and signal phase spectra, y(m)((m)+u(m)))


Common Filter Types

fs/2fH

A(f)

0

1

fs/2fH

A(f)

0

1

fs/2fL

A(f)

0

1

fs/2fH

A(f)

0

1

Low Pass

Band Pass

High Pass

Notch

fL fL


Filter Design

A(f). spectrum amplitude desired theproviding

)(

systemlinear for the b tscoefficien Find

:Problem

2

0

i

p

ii ikuby(k)


Filter Design Approach:

.,2

cos)(2

c

by c thefinding toproblem therecasts

)(

seriesFourier truncateda Using

2

0i

i

2

pipdff

iffA

f

ecfA

s

s

f

ss

p

p

f

ijf

i


Fourier Filter Coefficients

s

H

s

Li

s

LHs

s

L

s

Hi

s

LH

s

Li

s

Ls

s

Hi

s

H

f

if

f

if

ic

f

fffc

f

if

f

if

ic

f

ffc

f

if

ic

f

ffc

f

if

ic

f

fc

2sin

2sin

1,

)(2:Notch

2sin

2sin

1,

)(2:Bandpass

2sin

1,

2:passHigh

2sin

1,

2:pass Low

0

0

0

0


Gibbs Phenomena: Low Pass Filter

Gibbs Phenomenon

0

0.2

0.4

0.6

0.8

1

1.2

1 4 7 10 13 16 19 22 25 28 31

A(f)-box


Windowing Coefficients

p

iiw

p

iiw

p

iiw

p

iiw

cos46.054.0)( :Hamming

cos15.0)( :Hanning

||1)( :Welch

||1)( :Bartlett

2


Windowing Functions

Common Windowing Functions

0

0.2

0.4

0.6

0.8

1

1.2

1 4 7 10 13 16 19 22 25 28 31

Bartlett

Welch

Hamming

Hanning


Suppressing the Gibbs Phenomenon: Windowed Low Pass Filter

Window Effects Comparison

0

0.2

0.4

0.6

0.8

1

1.2

1 5 9 13 17 21 25 29 33

A(f)-box

A(f) windowed


FIR Filter Algorithm

p

ii

ipip

iip

f

ssi

ikubky

bb

iwciwb

dff

iffA

fc

s

2

0

th

2

0

)()(

Set 3.

t)coefficien windowi theis )(()(

table) theuse(2

cos)(2

compute p to0iFor 2.

0.pPick 1.


Two-Dimensional Fourier Transform

Previous signals were all one-dimensional There are common two-dimensional signals

—IMAGES! There are some differences

– Sampling interval reflects change of position– Spatial frequency reflects rate of change in pixel

values


Notation for 2-D Fourier Transforms

N-1

210

0 1 2 M-1…

k →

i → …

Δx

Δy

),(),(

bygiven are valuespixel The

).,( (image) signal

analog ldimensiona- twoaGiven

yixkzikz

yxz

a

a


Definition: 2-D DFT

NnMm

eeikznmZ M

mkj

N

nijM

k

N

i

0 and ,0 where

),(),(

:bygiven is

ansformFourier tr ldimensiona- twodiscrete, The

221

0

1

0


Definition: 2-D Inverse DFT

NiMk

eenmZMN

ikz M

mkj

N

nijM

k

N

i

0 and ,0 where

),(1

),(

:bygiven is transform

Fourier ldimensiona- twoinverse, discrete, The

221

0

1

0


Algorithm for 2-D FFT

MmeknYnmZ

N-n

NneikzknY

M-k

M

k

M

mkj

N

i

N

nij

0,),(),(

compute toalgorithm FFT use 1 to0For 2.

0,),(),(

compute toalgorithm FFT use 1 to0For 1.

1

0

2

1

0

2


2-D Spectral Definitions

2|),(|

: spectrumpower The

)),(Re(

)),(Im(arctan

: spectrum phase The

|),(|),(

: spectrum amplitude The

nmZP(m,n)

P(m,n)

nmZ

nmZ(m,n)

(m,n)

nmZnmA

A(m,n)


Interpreting the Results

yN

nf

xM

mf

Z(m,n)yx

yx

ynxm

and

sfrequencie spatialat responsefrequency spatial theis

Then ly.respective ,directions and in the

intervals sampling therepresent and Let


Example: A Structured Image

Construct a linear combination of 2-D sine functions

Render a 2-D power spectral density estimate

Observe the effects of ideal filtering on the power spectrum


Examples


Example: Bandpass Filter


System Identification

Given a linear system Model the input/output relationship

Black box system with unknown transfer function

Input u(k) Output y(k)


Modeling Basis

Tmn

m

jj

n

ii

bbbaaa

kjkubi)y(kay(k)

],...,,,...,[

by given vector parameter heconsider t and

0 ),(

by described is

system timediscretelinear a that Recall

10,2,1

01


Modeling Basis Re-Written

0),(

hence

)](),...,0(),(),...,1([)(

by defining and

)()(

obtain we Isolating

01

kkxy(k)

mkuunkykykx

x(k)

jkubi)y(kay(k)

y(k)

T

T

m

jj

n

ii


Another Simplifying Notation

vector parameter somefor

can write Then we

)1(

)1(

)0(

and

)1(

)1(

)0(

3.

0for 0 conditions initial The 2.

,0for 0 i.e., causal, is system The 1.

Assume

θXy

Ny

y

y

y

Nx

x

x

X

k y(k)

ku(k)

T

T

T


The System Identification Question

Can we find a parameter vector θ that minimizes the error e(θ) given bye(θ) ≡ y – Xθ?

Yes (!) but it takes some calculus. Consider the real valued, quadratic, error

expression J(θ) given by J(θ) ≡ eT(θ)e(θ) (note J(θ)=|e(θ)|2), differentiate, and find the critical value


Some Algebra

J(θ) = eT(θ)e(θ) = (y – Xθ)T (y – Xθ) = (yT – (Xθ)T) (y – Xθ) = (yT – θTXT) (y – Xθ) = yTy – yTXθ – θTXTy + θTXTXθ = yTy – 2yTXθ + θTXTXθ (Why?)

dJ(θ)/dθ = –2yTX+2θTXTX Hence dJ(θ)/dθ = 0 ↔ yTX = θTXTX


Some Algebra (continued)

dJ(θ)/dθ = 0 ↔ yTX = θTXTX ↔ yTX (XTX)-1 = θT

↔ θT = yTX (XTX)-1

↔ (θT)T = (yTX (XTX)-1)T

↔ θ = ((XTX)-1)T XT (yT)T

↔ θ = (XTX)-1 XT y Thus, if XTX is invertible and y is given we

can compute θ!


Conditions for XTX to be Invertible

The number of samples N must be at least as great as the number of unknown parameters n + (m + 1)

The magnitude spectrum of u(k) must be nonzero for at least n + (m+1) frequencies

introduction to fourier processing a spectrum of possibilities…

Documents