introduction to adaptive signal processing

EECS0712 Adaptive Signal Processing1

Introduction to Adaptive SignalProcessing

EECS0712 Adaptive Signal Processing1

Introduction to Adaptive SignalProcessing

Assoc. Prof. Dr. Peerapol YuvapoositanonDept. of Electronic Engineering

CESdSP ASP1-1EECS0712 Adaptive Signal Processing

http://embedsigproc.wordpress.com/eecs0712Assoc. Prof. Dr. P.Yuvapoositanon

Course Outline

• Introduction to Adaptive Signal Processing• Adaptive Algorithms Families:• Newton’s Method and Steepest Descent• Least Mean Squared (LMS)• Recursive Least Squares (RLS)• Kalman Filtering• Applications of Adaptive Signal Processing in

Communications and Blind Equalization

• Introduction to Adaptive Signal Processing• Adaptive Algorithms Families:• Newton’s Method and Steepest Descent• Least Mean Squared (LMS)• Recursive Least Squares (RLS)• Kalman Filtering• Applications of Adaptive Signal Processing in

Communications and Blind EqualizationCESdSP ASP1-2

EECS0712 Adaptive Signal Processinghttp://embedsigproc.wordpress.com/eecs0712

Assoc. Prof. Dr. P.Yuvapoositanon

Evaluation

• Assignment= 20 %• Midterm = 30 %• Final = 50 %

CESdSPEECS0712 Adaptive Signal Processing

ASP1-3

Textbooks

ASP1-4

http://embedsigproc.wordpress.com/eecs0712-adaptive-signal-processing/

ASP1-5

QR code

ASP1-6

Adaptive Signal Processing

• Definition: Adaptive signal processing is thedesign of adaptive systems for signal-processing applications.

[http://encyclopedia2.thefreedictionary.com/adaptive+signal+processing]

• Definition: Adaptive signal processing is thedesign of adaptive systems for signal-processing applications.

[http://encyclopedia2.thefreedictionary.com/adaptive+signal+processing]

ASP1-7

System Identification

• Let’s consider a system called “plant”• We need to know its characteristics, i.e., The

impulse response of the system

Plant Comparison

Error of Plant Outputs

Error of Estimation

• Error of estimation is represented by thesignal energy of error

d dy y

Adaptive System

• We can do it adaptively

• Adjust the weight for minimum error e

One-weight

2 20 0 0 0

( ) 2( )( ) ( )I I

d dy y

w x w x w x w x

CESdSP

2 20 0 0 0

( ) 2( )( ) ( )I I

d dy y

w x w x w x w x

ASP1-14EECS0712 Adaptive Signal Processing

Error Curve

• Parabola equation

Partial diff. and set to zero

• Partial differentiation

• Set to zero

• Result:

0 0 0 00 0

2 20 0

( ) 2( )( ) ( )

I II I

ew x w x w x w x

w x w x

• Partial differentiation

• Set to zero

• Result:

CESdSP

0 0 0 00 0

2 20 0

( ) 2( )( ) ( )

I II I

ew x w x w x w x

w x w x

2 20 00 2 2 Iw x w x

0 0Iw w

Multiple Weight Plants

• We calculate the weight adaptively• Questions:

– What is the type of signal “x” to be used, e.g.Sine, Cosine or Random signals ?

– If there is more than one weight w0 , i.e., w0….wN-

1, how do we calculate the solution?

• We calculate the weight adaptively• Questions:

– What is the type of signal “x” to be used, e.g.Sine, Cosine or Random signals ?

– If there is more than one weight w0 , i.e., w0….wN-

1, how do we calculate the solution?

Plants with Multiple Weight

• If we have multiple weights

CESdSP

10 1w w z w

• In the case of two-weight

Two-weight

• From

• We construct the x as vector with firstelement is the most recent

(3), (2), (1), (0), ( 1), ( 2),...x x x x x x

• From

• We construct the x as vector with firstelement is the most recent

CESdSP

[ (3) (2) (1) (0)...]Tx x x xx

Plants with Multiple Weight(aka “Transversal Filter”)

• If we have multiple weights( )x n ( 1)x n

CESdSP

0 ( )w x n0 ( 1)w x n

0 0( ) ( ) ( 1)y n w x n w x n

Regression input signal vector

• If the current time is n, we have “Regressioninput signal vector”

[ ( ) ( 1) ( 2) ( 3)...]Tx n x n x n x n x

CESdSP

[ ( ) ( 1) ( 2) ( 3)...]Tx n x n x n x n x

CESdSP

ˆ [ ]

II I T

Convolution

• Output of plant is a convolution

• Ex For N=2

( ) ( )N

y n w x n k

• Output of plant is a convolution

• Ex For N=2

CESdSP

( ) ( )N

y n w x n k

0 0( ) ( 0) ( 1)y n w x n w x n

(3) (3) (2)

(2) (2) (1)

(1) (1) (0)

(0) (0) ( 1)

( 1) ( 1) ( 2)

y w x w x

CESdSP

(3) (3) (2)

(2) (2) (1)

(1) (1) (0)

(0) (0) ( 1)

( 1) ( 1) ( 2)

y w x w x

• We can use a vector-matrix multiplication• For example, for n=3 we construct y(3) as

• For example, for n=1 we construct y(1) as

0 1 0 1

(3)(3) (3) (2) [ ] (3)

xy w x w x w w

• We can use a vector-matrix multiplication• For example, for n=3 we construct y(3) as

• For example, for n=1 we construct y(1) as

CESdSP

0 1 0 1

(3)(3) (3) (2) [ ] (3)

xy w x w x w w

0 1 0 1

(1)(1) (1) (0) [ ] (1)

xy w x w x w w

0 1 0 1

(3)(3) (3) (2) [ ] (3)

(2)(2) (2) (1) [ ] (2)

(1)(1) (1) (0) [ ] (1)

(2)(0) (0) ( 1) [ ] (0

xy w x w x w w

CESdSP

0 1 0 1

(3)(3) (3) (2) [ ] (3)

(2)(2) (2) (1) [ ] (2)

(1)(1) (1) (0) [ ] (1)

(2)(0) (0) ( 1) [ ] (0

xy w x w x w w

• The error squared is

• Let us stop there to consider Random signaltheory first.

ˆ ˆ( ) 2( )( ) ( )T T T T

d dy y

w x w x w x w x

• The error squared is

• Let us stop there to consider Random signaltheory first.

CESdSP

ˆ ˆ( ) 2( )( ) ( )T T T T

d dy y

w x w x w x w x

Review of Random Signals

Wireless Transmissions

• Ideal signal transmission

11 00 11 00 11 0011 11 11 000011

ASP2-30

11 00 11 00 11 0011 11 11 000011

Information

Information is Random

Random variable

Random Variable

• Random variable is a function• For a single time Coin Tossing

x HX x

• Random variable is a function• For a single time Coin Tossing

CESdSP

x HX x

Our signal x(n) is a RandomVariable

• For a series of Coin Tossing

x HX x

• For a series of Coin Tossing

CESdSP

x HX x

0 1 2 3 4{ , , , , ,....}x x x x x x

Coin tossing and Random Variable

• If random

• We have random variable X0 1 2 3 4

{ , , , , }

x H H T H T

x x x x x

CESdSP

• If random

• We have random variable X

0 1 2 3 4( ) { ( ), ( ), ( ), ( ), ( )}

{ ( ), ( ), ( ), ( ), ( )}

{1,1, 1,1, 1}

iX x X x X x X x X x X x

X H X H X T X H X T

Random Digital Signal

• If the random variable is a function of time, itis called a stochastic process

Probability Mass Function

• We need also to define the probability of eachrandom variable

( ) { ( ), ( ), ( ), ( ), ( )}

{1,1, 1,1, 1}

X x X H X H X T X H X T

CESdSP

( ) { ( ), ( ), ( ), ( ), ( )}

{1,1, 1,1, 1}

X x X H X H X T X H X T

Probability Mass Function

• PMF is for Discrete distribution function

Time and Emsemble

Probability of X(2)

Probability Density Function

• PDF is for Continuous Distribution Function

Probability Density Function

• PDF values can be > 1 as long as its area undercurve is 1

CESdSP

Cumulative Distribution Function

CESdSP

( ( )) Pr[ ( )]P x n X x n x

( ( )) ( )x n

P x n p z dz

CESdSP

( ( )) ( )x n

P x n p z dz

Expectation Operator

CESdSP

Expected Value

• Expected value is known as the “Mean”

{ } ( )X XE x xp x dx

CESdSP

{ } ( )X XE x xp x dx

Example of Expected Value(Discrete)

• We toss a die N times and get a set ofoutcomes

• Suppose we roll a die with N=6, we might get

{ ( )} { (1), (2), (3),..., ( )}X i X X X X N

• We toss a die N times and get a set ofoutcomes

• Suppose we roll a die with N=6, we might get

CESdSP

{ ( )} { (1), (2), (3),..., ( )}X i X X X X N

{ ( )} {2,3,6,3,1,1}X i

Example of Expected Value(Discrete)

• But, empirically we have Empirical (MonteCarlo) estimate as Expected Value

{ } ( )Pr( ( ))

1 1 1 11 2 3 6

3 6 3 62.67

E x X i X X i

CESdSP

{ } ( )Pr( ( ))

1 1 1 11 2 3 6

3 6 3 62.67

E x X i X X i

Theoretical Expected Value

• But in theory, for a die

{ } ( )Pr( ( ))

1 1 1 1 1 11 2 3 4 5 6

6 6 6 6 6 63.5

E X X i X X i

1Pr( ( ))

6X X i

CESdSP

{ } ( )Pr( ( ))

1 1 1 1 1 11 2 3 4 5 6

6 6 6 6 6 63.5

E X X i X X i

Ensemble Average

i ensembles

1 1 2 2Ensemble Average of (1) (1)Pr[ (1)] (1)Pr[ (1)]

(1)Pr[ (1)]N N

x x x x x

1 ensemble

ASP1-50

i ensembles

Ensemble Average

{ ( )}E x n

ASP1-51

{ ( )} ( ) ( ( )) ( )E x n x n p x n dx n

{ ( )}E x n

• I) Linearity

ASP1-52

{ ( ) ( )} { ( )} { ( )}E ax n by n aE x n bE y n

• II)

{ ( ) ( )} { ( )} { ( )}E x n y n E x n E y n

ASP1-53

{ ( ) ( )} { ( )} { ( )}E x n y n E x n E y n

• III)

{ ( )} ( ( )) ( ( )) ( )E y n g x n p x n dx n

ASP1-54

{ ( )} ( ( )) ( ( )) ( )E y n g x n p x n dx n

Autocorrelation

1 1( , ) { ( ) ( )}r n m E x n x mxx

ASP1-55

1 11 1 1 1 1 1( , ) ( ) ( ) ( ( ), ( )) ( ) ( )r n m x n x m p x n x m dx n x m

xx x x

1 1(1, 4) { (1) (4)}r E x xxx

ASP1-56

Autocorrelation

• n=m

2( , ) ( , ) { ( )}r n m r n n E x n xx xx

ASP1-57

2( , ) ( , ) { ( )}r n m r n n E x n xx xx

Autocorrelation Matrix

(0,0) (0,1) (0, 1)

(1,0) (1,1) (1, 1)

( 1,0) ( 1,1) ( 1, 1)

r r r N

r N r N r N N

xx xx xx

xx xx xxxx

xx xx xx

ASP1-58

(0,0) (0,1) (0, 1)

(1,0) (1,1) (1, 1)

( 1,0) ( 1,1) ( 1, 1)

r r r N

r N r N r N N

xx xx xx

xx xx xxxx

xx xx xx

Covariance

( , ) {[ ( ) ( )][ ( ) ( )]}c n m E x n n x m m xx

ASP1-59

( , ) {[ ( ) ( )][ ( ) ( )]}c n m E x n n x m m xx

Stationarity (I)

• I)

{ ( )} { ( )}E x n E x m n1

ASP1-60

Stationarity (II)

• II)

( , ) { ( ) ( )}r n n m E x n x n m xx

1 1 1 1( , ) { ( ) ( )}r n n m E x n x n m xx

ASP1-61

1 1 1 1( , ) { ( ) ( )}r n n m E x n x n m xx

Expected Value of Error Energy

• Let’s take the expected value of error energy

2 2 2ˆ ˆ{ } {( ) 2( )( ) ( ) }

ˆ ˆ ˆ{( )( )} 2 {( )( )} {( )( )}

ˆ ˆ ˆ{ } 2 {( )( )} { }

ˆ ˆ ˆ2 {( )( )}

T T T T

T T T T T T

T T T T

w x w x w x w x

w x x w x w w x w x x w

w xx w x w x w w xx w

w Rw x w x w w Rw

ASP1-62

2 2 2ˆ ˆ{ } {( ) 2( )( ) ( ) }

ˆ ˆ ˆ{( )( )} 2 {( )( )} {( )( )}

ˆ ˆ ˆ{ } 2 {( )( )} { }

ˆ ˆ ˆ2 {( )( )}

T T T T

T T T T T T

T T T T

w x w x w x w x

w x x w x w w x w x x w

w xx w x w x w w xx w

w Rw x w x w w Rw

Vector-Matrix Differentiation

ˆI)ˆ

ˆ ˆ ˆII) 2ˆ

w x xw

w xx w Rww

CESdSP

ˆI)ˆ

ˆ ˆ ˆII) 2ˆ

w x xw

w xx w Rww

Partial diff. and set to zero

• Differentiation

• Result:

ˆ0 2 {( ) } 2ˆ

ˆ2 { } 2

w x x Rww

• Differentiation

• Result:

CESdSP

ˆ0 2 {( ) } 2ˆ

ˆ2 { } 2

w x x Rww

1ˆ w R rASP1-64

EECS0712 Adaptive Signal Processinghttp://embedsigproc.wordpress.com/eecs0712

Assoc. Prof. Dr. P.Yuvapoositanon

2-D Error surface

CESdSP

1ˆ w R r

Four Basic Classes of AdaptiveSignal Processing

• I) Identification• II) Inverse Modelling• III) Prediction• IV) Interference Cancelling

ASP1-66

• I) Identification• II) Inverse Modelling• III) Prediction• IV) Interference Cancelling

The Four Classes of AdaptiveFiltering

ASP1-67

System Identification

ASP2-68

Inverse Modelling

ASP2-69

Prediction

ASP2-70

Interference Canceller

ASP2-71

What are we looking for inAdaptive Systems?

• Rate of Convergence• Misadjustment• Tracking• Robustness• Computational Complexity• Numerical Properties

ASP1-72

introduction to adaptive signal processing

Education

signal-processing and adaptive prototissue formation in

initial inputs: adaptive front-end signal processing

adaptive signal processing , ebook . ingle

adaptive signal processing & machine intelligence … ›...

a novel approach for adaptive signal processing novel...

week 4ele 774 - adaptive signal processing 1 linear...

adaptive signal processing

a study of implementation of digital signal processing for...

least squaresele 774 - adaptive signal processing 1 method...

adaptive signal and image processing

contributions to robust adaptive signal processing with...

adaptive signal processing · 2019-02-28 · motivation...

solutionsmanual-statistical and adaptive signal processing

rosemount 8800, rosemount 8800 vortex adaptive digital...

signal processing with adaptive sparse structured ... ›...

novel adaptive signal processing techniques for underwater...

universit`a degli studi di parma adaptive signal processing

high-level adaptive signal processing · acoustics, speech,...

ele 674/774 adaptive signal...

eecs0712 adaptive signal processing 2 introduction to...