EGR653 Digital and Adaptive Systems

- 1 -

Active Learning Exercise #1

The LS Optimal Filter: The System Identification Problem

The System Identification Problem

Consider Figure 1, where an input signal [ ]kx drives an unknown system to produce an

output [ ]kd . We intend to use these input and output signals to derive an optimal filter

that models the unknown system. The difference between the true filter output (which

may be corrupted by an additional noise source as shown in the figure) and the estimated

filter output [ ]ky is the error signal [ ]ke . We will derive the filter model to minimize the

average squared error [ ]{ }keE 2 .

Σ[ ]kxfilter

unknown

estimate

filter

[ ]kd[ ]kn

Σ [ ]ke[ ]ky −

Figure 1: The System Identification Problem

The Normal Equations

The minimization of the MSE in an optimal FIR filtering scenario results in a set of linear

equations. These equations, known as the Normal Equations, can be solved to yield the

best fit filter estimate in a Least Squares sense. The Normal Equations are expressed as

gR =θφφ

v,

where

[ ] [ ] [ ][ ] [ ] [ ]

[ ] [ ]

+−

−−

=

0

101

10

xxxx

xxxxxx

xxxxxx

RNR

NRRR

NRRR

R

L

OM

L

L

φφ ,

[ ] [ ] [ ][ ]Tdxdxdx NRRRg −−= L10 ,

with [ ] [ ] [ ]{ }knxnxEkRxx += , [ ] [ ] [ ]{ }knxndEkRdx += and θv

is the 1+N tap parameter

vector. The solution of the Normal Equations results in the LS optimal parameter vector

estimate

( ) gRLS

1* −= φφθ

v.

EGR653 Digital and Adaptive Systems

- 2 -

Procedure

1. Create a MATLAB script to simulate the system identification problem (at your

option, you may use Simulink). Choose some suitable FIR filter and filter order as

your “true” system response, denoted as trueθv

. For input [ ]kx , create a (very) long

vector of AWGN generated with the randn command (seed based on the current

clock). Next, generate the desired signal [ ]kd using your true filter trueθv

and [ ]kx via

the filter command. At this stage, do not add any noise to [ ]kd . Construct the

Normal Equations by estimating [ ]kRxx and [ ]kRdx using the xcorr command. Solve

the Normal Equations to arrive at your optimal filter estimate *

LSθv

. Compute the

normalized parameter error as (use the MATLAB command norm, which computes

the 2-norm):

true

LStrue

θ

θθv

vv*−

.

Furthermore, compute the error signal [ ]ke . In this situation, with no added noise, the

normalized parameter error and the variance of the error signal [ ]ke should both be

very small. If this is the case, your equations are correctly configured and you may

move to the next step.

2. We now consider algorithm performance under conditions of noise interference.

Assume that AWGN [ ]kn is now added to the desired signal [ ]kd , where the SNR is

defined as

2

2

n

xSNRσ

σ= .

Above, 2

xσ is the variance of the input signal [ ]kx and 2

nσ is the variance of the

additive noise [ ]kn . Generate a plot of normalized parameter error for a range of

dBSNRdB 1010 ≤≤− . On the same graph, plot the variance of the error signal [ ]ke

for each SNR.

3. Discuss the ability of the Normal Equations to predict the LS optimal filter

coefficients as the SNR varies. Furthermore, how does the variance of the error signal

change with SNR? Is this as expected?

Exercises

1. Clearly describe the equivalence of solving an overdetermined set of equations using

the pseudo-inverse and the solution of the Normal Equations using sample-based

estimates of the autocorrelation functions.

2. Considering the cost function [ ]kJ , show that the input regressor vector is orthogonal

to the optimal error, that is,

[ ] [ ]{ } 0* =kekE φ .

EGR653 Digital and Adaptive Systems

- 3 -

3. The correlation matrix φφR is Toeplitz, and hence, positive semi-definite (all non-

negative eigenvalues). Show that 0~~

≥θθ φφRT .

4. In the simulation, we used AWGN as the input signal [ ]kx . This spectrally flat

wideband signal is an excellent choice for system identification. In the derivation of

optimal filters, it is required that the input signal contain sufficient “spectral

excitation” or else the resulting derived filter will be suboptimal. Discuss why you

think an input with sufficient spectral excitation is necessary. Contrast your

discussion with why you think using a sinusoid as the input signal is suboptimal.

Turn in (due at start of class next week):

• A printout of your m-file script, well commented and neatly formatted that

implements the entire testing range for Procedure step (2). Include a comment

block to identify your name, script name, date and course, instructor, description,

etc. Submit this script to Blackboard prior to class.

• A hardcopy of your plot from Procedure step (2).

• Discussion for Procedure step (3).

• Fully detailed solutions to the Exercise problems (hard copy).