ece 531: detection and estimation theory example of detection
TRANSCRIPT
ECE 531: Detection and Estimation Theory
Natasha [email protected]://www.ece.uic.edu/~devroye
Spring 2011
Example of detection
Example of estimation
Goals
• infer value of unknown state of nature based on noisy observations
Mathematically, optimally
model of hypothesis H
NatureTransmission /
measurementProcessing
Noise
phenomenon
experiment
the ``truth’’
model of observationor transmission
process
decision ruleestimation function
mapping from observation space
to decisions/estimates
Detection example 1: digital communications
Source Encoder Channel Decoder Destination
Noise
10001010100010
Detect?
Detection example 2: Radar communication
Send
Receive
Hypothesis
Hypothesis
Detect?
Further examples
• Sonar: enemy submarine
• Image processing: detect an aircraft from infrared images
• Biomedicine: cardiac arryhthmia from heartbeat sound wave
• Control: detect occurrence of abrupt change in system to be controlled
• Seismology: detect presence of oil deposit
Difference between detection and estimation?
• Detection:
• Estimation:
Discrete set of hypotheses
Right or wrong
Continuos set of hypotheses
Almost always wrong - minimize error instead
Estimation example 1: communications
• Pulse amplitude modulation (PAM)
Analog source Sampler Transmitter
Receiver?
Estimation example 2: Radar
Send
Receive
Hypothesis
Hypothesis
Estimate?
Our methods
• Will treat everything generally, with a unified mathematical representation
• Bias towards Gaussian noise and linear observation - parameter models
• Examples mainly drawn from communications / radar
Aside: “Classical” vs. “Bayesian”
• Hypotheses/parameters are fixed, non-random
• Hypotheses/parameters are treated as random variables with assumed priors (or a priori distributions)
Classical
Bayesian
Course outline
ECE 531: Detection and Estimation
University of Illinois at Chicago, ECE
Spring 2010
Instructor: Natasha Devroye, [email protected]
Course coordinates: Tuesday, Thursday from 2-3:15pm in TH 208 (Taft Hall).
Office hours: Monday from 4-5:30pm in SEO 1039, or by appointment
Welcome to ECE 531! This course is a graduate-level introduction to detection and estimation theory,
whose goal is to extract information from signals in noise. A solid background in probability and
some knowledge of signal processing is needed.
Course Textbook: Fundamentals of Statistical Signal Processing, Volume 1: Estimation Theory, by
Steven M. Kay, Prentice Hall, 1993 and (possibly) Fundamentals of Statistical Signal Processing,
Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998.
Other useful references:
Harry L. Van Trees, Detection, Estimation, and Modulation Theory, Part I, II, III, IV
H. Vincent Poor, Introduction to Signal Detection and Estimation
Louis L. Scharf and Cedric Demeure, Statistical Signal Processing: Detection, Estimation, and Time
Series Analysis
Carl Helstrom, Elements of Signal Detection and Estimation. It's out of print, so here's my pdf copy.
Notes: I will follow the course textbooks fairly closely, using a mixture of slides (highlighting the
main points and with nice illustrations) and more in-depth blackboard derivations/proofs in class. I
will post a pdf version of the slides as they become ready here, but the derivations will be given in
class only.
Topics: Estimation Theory:
General Minimum Variance Unbiased Estimation, Ch.2, 5
Cramer-Rao Lower Bound, Ch.3
Linear Models+Unbiased Estimators, Ch.4, 6
Maximum Likelihood Estimation, Ch.7
Least squares estimation, Ch.8
Bayesian Estimation, Ch.10-12
Detection Theory:
Statistical Detection Theory, Ch.3
Deterministic Signals, Ch.4
Random Signals, Ch.5
Statistical Detection Theory 2, Ch.6
Non-parametric and robust detection
Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 =
max(Exam 2, Final) (20%), Project (15%), Final exam (30%).
http://www.ece.uic.edu/~devroye/courses/ECE531/
1 of 3 1/11/10 8:50 PM
Fundamentals of Statistical Signal Processing, Volume 1: Estimation Theory, by Steven M. Kay, Prentice Hall, 1993
Fundamentals of Statistical Signal Processing, Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998.
ECE 531: Detection and Estimation
University of Illinois at Chicago, ECE
Spring 2010
Instructor: Natasha Devroye, [email protected]
Course coordinates: Tuesday, Thursday from 2-3:15pm in TH 208 (Taft Hall).
Office hours: Monday from 4-5:30pm in SEO 1039, or by appointment
Welcome to ECE 531! This course is a graduate-level introduction to detection and estimation theory,
whose goal is to extract information from signals in noise. A solid background in probability and
some knowledge of signal processing is needed.
Course Textbook: Fundamentals of Statistical Signal Processing, Volume 1: Estimation Theory, by
Steven M. Kay, Prentice Hall, 1993 and (possibly) Fundamentals of Statistical Signal Processing,
Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998.
Other useful references:
Harry L. Van Trees, Detection, Estimation, and Modulation Theory, Part I, II, III, IV
H. Vincent Poor, Introduction to Signal Detection and Estimation
Louis L. Scharf and Cedric Demeure, Statistical Signal Processing: Detection, Estimation, and Time
Series Analysis
Carl Helstrom, Elements of Signal Detection and Estimation. It's out of print, so here's my pdf copy.
Notes: I will follow the course textbooks fairly closely, using a mixture of slides (highlighting the
main points and with nice illustrations) and more in-depth blackboard derivations/proofs in class. I
will post a pdf version of the slides as they become ready here, but the derivations will be given in
class only.
Topics: Estimation Theory:
General Minimum Variance Unbiased Estimation, Ch.2, 5
Cramer-Rao Lower Bound, Ch.3
Linear Models+Unbiased Estimators, Ch.4, 6
Maximum Likelihood Estimation, Ch.7
Least squares estimation, Ch.8
Bayesian Estimation, Ch.10-12
Detection Theory:
Statistical Detection Theory, Ch.3
Deterministic Signals, Ch.4
Random Signals, Ch.5
Statistical Detection Theory 2, Ch.6
Non-parametric and robust detection
Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 =
max(Exam 2, Final) (20%), Project (15%), Final exam (30%).
http://www.ece.uic.edu/~devroye/courses/ECE531/
1 of 3 1/11/10 8:50 PM
Kalman filtering
Estimation: General Minimum Variance Unbiased Estimation
• Bias: (expected value of estimator - true value of data)
• MVUE:
Estimation: Cramer-Rao lower bound
• Lower bound on variance of ANY unbiased estimator!
• Usage:
• assert whether an estimator is MVUE
• benchmark against which to measure the performance of an unbiased estimator
• feasibility studies
• Depends on?
NatureTransmission /
measurementProcessing
Noise
Estimation: linear models
• What’s a linear model and why is it useful?
• What can be said?
• Best Linear Unbiased Estimators (BLUE)
Estimation: Maximum Likelihood Estimation
• Alternative to MVUE which is hard to find in general
• Easy to compute - very widely used and practical
• What is the MLE?
• Properties?
Estimation: Least Squares
• Alternative estimator with no general optimality properties, but nice and intuitive and no probabilistic assumptions on data are made - only need a signal model
• Advantages?
• Disadvantages?
Estimation: Bayesian Estimation
• Parameter to be estimated is assumed to be random, according to some prior distribution which models our knowledge of it
• Bayesian Minimum Mean Squared Error (MMSE):
• Applications to Gaussian noise / linear model
Estimation: Bayesian Estimation
• General risk functions - arbitrary “cost” functions
• Maximum a posteriori (MAP) estimation
• Linear MMSE: constrain estimator to be linear - very practical
Estimation: Kalman filtering
• recursive filter for estimating internal state of linear dynamical system from a series of noisy measurements
• example: tracking a moving target using radar measurements
linear dynamical system
estimate the position and/or velocity
noisy measurements
Estimation: Kalman filtering
• recursive filter for estimating internal state of linear dynamical system from a series of noisy measurements
• than can recursively estimate/predict and update the state covariances as:
Course outline
ECE 531: Detection and Estimation
University of Illinois at Chicago, ECE
Spring 2010
Instructor: Natasha Devroye, [email protected]
Course coordinates: Tuesday, Thursday from 2-3:15pm in TH 208 (Taft Hall).
Office hours: Monday from 4-5:30pm in SEO 1039, or by appointment
Welcome to ECE 531! This course is a graduate-level introduction to detection and estimation theory,
whose goal is to extract information from signals in noise. A solid background in probability and
some knowledge of signal processing is needed.
Course Textbook: Fundamentals of Statistical Signal Processing, Volume 1: Estimation Theory, by
Steven M. Kay, Prentice Hall, 1993 and (possibly) Fundamentals of Statistical Signal Processing,
Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998.
Other useful references:
Harry L. Van Trees, Detection, Estimation, and Modulation Theory, Part I, II, III, IV
H. Vincent Poor, Introduction to Signal Detection and Estimation
Louis L. Scharf and Cedric Demeure, Statistical Signal Processing: Detection, Estimation, and Time
Series Analysis
Carl Helstrom, Elements of Signal Detection and Estimation. It's out of print, so here's my pdf copy.
Notes: I will follow the course textbooks fairly closely, using a mixture of slides (highlighting the
main points and with nice illustrations) and more in-depth blackboard derivations/proofs in class. I
will post a pdf version of the slides as they become ready here, but the derivations will be given in
class only.
Topics: Estimation Theory:
General Minimum Variance Unbiased Estimation, Ch.2, 5
Cramer-Rao Lower Bound, Ch.3
Linear Models+Unbiased Estimators, Ch.4, 6
Maximum Likelihood Estimation, Ch.7
Least squares estimation, Ch.8
Bayesian Estimation, Ch.10-12
Detection Theory:
Statistical Detection Theory, Ch.3
Deterministic Signals, Ch.4
Random Signals, Ch.5
Statistical Detection Theory 2, Ch.6
Non-parametric and robust detection
Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 =
max(Exam 2, Final) (20%), Project (15%), Final exam (30%).
http://www.ece.uic.edu/~devroye/courses/ECE531/
1 of 3 1/11/10 8:50 PM
Fundamentals of Statistical Signal Processing, Volume 1: Estimation Theory, by Steven M. Kay, Prentice Hall, 1993
Fundamentals of Statistical Signal Processing, Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998.
ECE 531: Detection and Estimation
University of Illinois at Chicago, ECE
Spring 2010
Instructor: Natasha Devroye, [email protected]
Course coordinates: Tuesday, Thursday from 2-3:15pm in TH 208 (Taft Hall).
Office hours: Monday from 4-5:30pm in SEO 1039, or by appointment
Welcome to ECE 531! This course is a graduate-level introduction to detection and estimation theory,
whose goal is to extract information from signals in noise. A solid background in probability and
some knowledge of signal processing is needed.
Course Textbook: Fundamentals of Statistical Signal Processing, Volume 1: Estimation Theory, by
Steven M. Kay, Prentice Hall, 1993 and (possibly) Fundamentals of Statistical Signal Processing,
Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998.
Other useful references:
Harry L. Van Trees, Detection, Estimation, and Modulation Theory, Part I, II, III, IV
H. Vincent Poor, Introduction to Signal Detection and Estimation
Louis L. Scharf and Cedric Demeure, Statistical Signal Processing: Detection, Estimation, and Time
Series Analysis
Carl Helstrom, Elements of Signal Detection and Estimation. It's out of print, so here's my pdf copy.
Notes: I will follow the course textbooks fairly closely, using a mixture of slides (highlighting the
main points and with nice illustrations) and more in-depth blackboard derivations/proofs in class. I
will post a pdf version of the slides as they become ready here, but the derivations will be given in
class only.
Topics: Estimation Theory:
General Minimum Variance Unbiased Estimation, Ch.2, 5
Cramer-Rao Lower Bound, Ch.3
Linear Models+Unbiased Estimators, Ch.4, 6
Maximum Likelihood Estimation, Ch.7
Least squares estimation, Ch.8
Bayesian Estimation, Ch.10-12
Detection Theory:
Statistical Detection Theory, Ch.3
Deterministic Signals, Ch.4
Random Signals, Ch.5
Statistical Detection Theory 2, Ch.6
Non-parametric and robust detection
Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 =
max(Exam 2, Final) (20%), Project (15%), Final exam (30%).
http://www.ece.uic.edu/~devroye/courses/ECE531/
1 of 3 1/11/10 8:50 PM
Detection: Statistical Detection Theory
• Binary hypothesis testing
Hypothesis
Hypothesis
Detection: Statistical Detection Theory
• Binary hypothesis testing
Detection: Statistical Detection Theory
PFA
PD
Detection: Deterministic Signals
• How to detect known signals in noise?
• The famous matched filter!
Xx[n]
T(x)
s[n]
• Generalized matched filter
• > 2 hypotheses
Detection: Random Signals
• What if s[n] is random?
• Key idea behind estimator-correlator:
• Linear model simplifies things again...
Detection: Statistical Decision Theory II
• model for the pdfs under 2 hypotheses are unknown
• Uniformly most powerful test
• Generalized likelihood ratio test
• Bayesian approach
• Wald test
• Rao test
Course structure
http://www.ece.uic.edu/~devroye/courses/ECE531/