ele 674/774 adaptive signal processingusezen/ele674/ele674_intro-p2.pdf · 2015. 9. 15. · s....

15/09/2015

1

ELE 774 - Adaptive Signal Processing 1

ELE 674/774 Adaptive Signal Processing

Dr. Umut Sezenwww.ee.hacettepe.edu.tr/~usezen/ele774/

Textbook S. Haykin, Adaptive Filter Theory, 4th Ed.,

Prentice Hall, 2002

Additional books:

B. Widrow and S. D. Stearns, Adaptive Signal Processing, Pearson, 1985.

A. D. Poularikas and Z. M. Ramadan, Adaptive Filtering Primer with MATLAB, Taylor & Francis, 2006.

ELE 674/774 - Adaptive Signal Processing 2

15/09/2015

2


Course Content (Tentative)

1. IntroductionBACKGROUND REVIEW2. Discrete-time and random signals3. Mathematical ToolsOPTIMAL LINEAR FILTERS4. Wiener Filters5. Linear PredictionADAPTIVE FILTERING6. Stochastic Gradient Descent Algorithms7. Family of LMS Algorithms8. Method of Least Squares9. Recursive Least Squares (RLS) Algorithm10. Square-Root Algorithms11. LMS and RLS Algorithms: Practical Issues12. Kalman Filtering (?)APPLICATIONS13. Spectrum Estimation14. Array Processing


Definition of filtering

A filter is commonly used to refer to a system that is designed to extract information about a prescribed quantity of interestfrom noisy data.

15/09/2015

3


Applications

Communications; radar, sonar,

Control Systems; navigation,

Speech/Image Processing; echo and noise cancellation, biomedical engineering

Others; seismology, financial engineering, etc.


!!! Noise and errors are statistical in nature !!!We will use statistical tools.

Applications

15/09/2015

4


Basic Kinds of Estimation

Filtering(real-time operation)

Smoothing(off-line operation)

Prediction(real-time operation)


Filter

Linear Non-LinearA filter is linear if the filtered,smoothed or predicted quantityat the output of the filter is a linear function of the observationsapplied to the filter input.

Otherwise, it is non-linear.

Filteru(t)

u(n)

y(t)

y(n)

Linear

Non-linear!!! Non-linear filters may be hard toAnalyse, if not impossible !!!

or

or

15/09/2015

5


Optimum Filter

Definition: Solution of an optimization problem wrt. a certain criterion with statistical parameters.

Nonlinear:

Maximum Likelihood (ML) sense (very difficult to implement)

Linear:

Minimum Mean Square Error (MMSE) sense Wiener filters, (Stationary environment)

Kalman filters, (Non-stationary environment)

Etc. (Any other criterion, e.g. ZF)


Adaptive Filters

Wiener Filter requires

Adaptive filtering can overcome these disadvantages! Recursive algorithm

No complete a priori information required Algorithm develops this information with increasing # of iterations.

If the environment is stationary → converges to the Wiener soln.

non-stationary → tracks the changes.

-a priori information of several statistics-estimation (knowledge of the system)

is needed before filtering-inversion of a huge matrix

-computationally inefficient!

15/09/2015

6


Analysis of Adaptive Filters

Rate of convergence (to the optimum Wiener soln.)

Misadjustment (deviation from the optimum Wienersoln.)

Tracking (the variations in a non-stationary environment)

Robustness (to disturbances of small energy)

Computational Requirements/Cost

Numerical Properties (Numerical stability & accuracy)


Linear Filter Structure

Structure:

FIR (Finite-duration Impulse Response) Transversal Filter (Tapped-delay line)

Lattice

Systolic array

IIR (Infinite-duration Impulse Response)

15/09/2015

7


Transversal Filter

multiplier

adder

unit-delayelement

ConvolutionSum


Transversal Filter

xH: Hermitian transpose

xH=(xT)*

15/09/2015

8


Lattice Predictor


Lattice Predictor

Predictor order (# of stages): M Forward prediction error

Backward prediction error

m: the m-th reflection coefficient Input seq. u(n) is correlated, backward prediction error

b(n) is uncorrelated Together with m, b(n) approximates d(n) (innovations

process).

15/09/2015

9

ELE 774 - Adaptive Signal Processing 17y3

Systolic Array Boundary cell

Internal cell

s


Systolic Array

Represents a parallel computing network

Used for efficient pipelined operation Matrix multiplication

Triangularisation

Back substitution (Matrix eqn. solving)

Example 3x3 matrix/3x1 vector

15/09/2015

10


IIR Filter

May have stability problems,We will prefer FIR filters for Adaptive filtering.


Adaptive Filtering Algorithms

Error Difference between

the filter output

a desired response

Mean Square Error

Weighted Error Squares

+

-

ε(n) : Errory(n)

d(n)

Filter

15/09/2015

11


Adaptive Filtering Algorithms

Stochastic Gradient Approach Cost Function depends on Mean-Square Error criterion

(!!! Stochastic, depends on second order statistics !!!)

Solution of Wiener-Hopf Equations Results in Wiener soln. but with an iterative approach

Based on Method of Steepest Descent

Use instantaneous values instead of expectations (LMS)

RequiresExpectations

E{.}


Adaptive Filtering Algorithms Least-Squares Estimation

Cost Function depends on sum of weighted error squares

Low computational complexity due to recursive operation

Three categories Standard RLS

Relies on Matrix Inversion Lemma

Numerically unstable, high computational complexity

Square-root RLS algorithm Based on QR-decomposition

Numerically stable

Fast RLS algorithm Exploits certain matrix structures to reduce complexity.

15/09/2015

12


Applications Four Classes

Identification system identification layered earth modeling

Inverse modeling deconvolution adaptive and blind

equalisation Prediction

linear predictive coding adaptive differential PCM spectrum analysis signal detection

Interference cancellation noise canceling echo cancellation adaptive beamforming


System Identification

Observing the output of aplant (system), given the inputsignal, tries to estimate theIR of the plant.

Filter coefficient are found byan adaptive algorithm.

15/09/2015

13


Adaptive Equalization

Removes intersymbolinterference (ISI).

Filter coefficients are foundby an adaptive algorithm.


Adaptive Spectrum Estimation

Parametric (AR) model

Linear AR filter input: white noise

output: observed signal

aim: find the model parameters by an adaptive algorithm.

15/09/2015

14


Adaptive Noise Cancellation

Electrocardiography (ECG)

Acoustic noise in speech

Active noise cancellation (headphones)


Echo Cancellation Coupling due to imperfect

balancing in hybrid transformer creates an echo in analog telephone lines.

Echo signal can be estimated by an adaptive filter and the subtracted out.

15/09/2015

15


Adaptive Beamforming

Multiple sensors (antenna, microphone, etc) used to steer the beam to a specific position.

Radar, sonar

Commun. systems,

Astrophysical exploration,

Biomedical signal processing, etc.


Historical Notes

To understand a science it is necessary to know its history. Auguste Comte (1798-1857)

Linear Estimation Theory Method of least squares, Gauss, 1795 Minimum mean square error estimation, late 1930s Discrete-time Wiener-Hopf equation, Levinson, 1947 Kalman filter, Swerling, 1958 and Kalman, 1960

Stochastic gradient algorithms, late 1950s Stochastic approximation, Robins and Monro, 1951 LMS algorithm, Widrow and Hoff, 1959 Gradient adaptive lattice (GAL) algorithm, Griffiths, 1977-8

15/09/2015

16


Historical Notes

Recursive Least Squares Algorithm Kalman filter, Godard Algorithm, Godard, 1974

Relationship between RLS and Kalman, Sayed and Kailath, 1994

QR decomposition based systolic array, Gentleman & Kung, 1981

Fast RLS algorithm, 1970s, Morf

Neural Networks Logical calculus for neural networks, McCulloch and Pitts, 1943

Perceptron, Rosenblatt, 1958

Back-propagation algorithm, Rumelhart, et al., 1986

Radial basis function network, Broomhead and Lowe, 1988


Applications

Adaptive Equalisation, 1960s Zero-forcing equaliser, Lucky, 1965 MMSE equaliser, Gersho, 1969, Proakis&Miller, 1969 Godard Algorithm, Godard, 1974 Fractionally Spaced Equaliser (FSE), Brady, 1970 Decision Feedback Equaliser (DFE), Austin 1967, MMSE,

Monsen, 1971.

Speech Coding Maximum Likelihood speech prediction, Saito and Itakura, 1966 Linear Predictive Coding (LPC), Atal and Hanauer 1970-1 Adaptive Lattice Predictor, Nakhoul and Cossell, 1981

15/09/2015

17


Applications

Spectrum Analysis, early 1900s Maximum entropy method, Burg, 1967

Method of multiple windows, Thomson, 1982

Adaptive noise cancellation, started at 1965

Adaptive Beamforming Intermediate Frequency (IF) sidelobe canceller, Howells, 1950

Control law for adaptive array antenna, Applebaum, 1966

Application of LMS, Widrow et al., 1967

Minimum Variance Distortionless Response (MVDR) beamformer, Capon, 1969

ele 674/774 adaptive signal processingusezen/ele674/ele674_intro-p2.pdf · 2015. 9. 15. · s....

Documents