correlation functions in time and frequency domain
TRANSCRIPT
-
8/10/2019 Correlation Functions in Time and Frequency Domain
1/48
2/11/2010
DelftUniversity ofTechnology
System Identification
&
Parameter Estimation
Correlation functions in time & frequency domain
Alfred C. Schouten, Dept. of Biomechanical Engineering (BMechE), Fac. 3mE
Wb2301: SIPE lecture 2
-
8/10/2019 Correlation Functions in Time and Frequency Domain
2/48
2SIPE, lecture 2 | xx
Contents
Correlation functions in time domain
Autocorrelations (Lecture 1)
Cross-correlations
General properties of estimators
Bias / variance / consistency
Estimators for correlation functions
Fourier transform
Correlation functions in frequency domain
Auto-spectral density
Cross-spectral density
Estimators for spectral densities
-
8/10/2019 Correlation Functions in Time and Frequency Domain
3/48
3SIPE, lecture 2 | xx
Lecture 1: (auto)correlation
functions
Autocorrelation function
Autocovariance
function
Autocorrelation coefficient
( ) ( ) 2 = = ( ) ( ) ( ) ( )xx x x xx xC E x t x t
[ ] = ( ) ( ) ( )xx E x t x t
2
2 0
= = =
( ) ( ) ( ) ( )( ) ( )
x x xx x xxxx
x x x xx
x t x t Cr E C
-
8/10/2019 Correlation Functions in Time and Frequency Domain
4/48
4SIPE, lecture 2 | xx
Lecture 1: autocovariance
function
Autocovariance
function:
If = 0, then autocovariance
and autocorrelation functions are identical
At zero lag:
( )( )
[ ] [ ] [ ] 2
2 2 2
2
=
= +
= +
=
( ) ( ) ( )
( ) ( ) ( ) ( )
( )
( )
xx x x
x x x
xx x x x
xx x
C E x t x t
E x t x t E x t E x t E
( )2 20( ) ( )xx x xC E x t
= =
-
8/10/2019 Correlation Functions in Time and Frequency Domain
5/48
5SIPE, lecture 2 | xx
Example autocovariances
Matlab
demo: Lec2_example_Cuu.m
0 1 2 3 4 5
-5
0
5
white
signal
-2 -1 0 1 2
-2
0
2Cuu
0 1 2 3 4 5-1
0
1
low-pass
-2 -1 0 1 2-0.05
0
0.05
0 1 2 3 4 5-1
0
1
sinusoid
-2 -1 0 1 2-0.5
0
0.5
0 1 2 3 4 5-5
0
5
sin.
+noise
time [s]
-2 -1 0 1 2-2
0
2
tau [s]
-
8/10/2019 Correlation Functions in Time and Frequency Domain
6/48
6SIPE, lecture 2 | xx
2D Probability density function
{ }dyytyydxxtxxdxdyyxf yx ++
-
8/10/2019 Correlation Functions in Time and Frequency Domain
7/48
7SIPE, lecture 2 | xx
2D Probability density function
Probability for certain values of y(t) given certain values of x(t)
Co-variance of y(t) with x(t): y(t) is related with x(t)
No co-variance between y(t) and x(t): 2D probability density functionis circular
Covariance between y(t) and x(t): 2D probability density function is
ellipsoidal
No co-variance between y(t) and x(t):
y(t) and x(t) are independent
no relation exist
transfer function is zero !
-
8/10/2019 Correlation Functions in Time and Frequency Domain
8/48
8SIPE, lecture 2 | xx
Cross-correlation functions
Measure for common structure in two signals:
Cross-correlation
Cross-covariance
Cross-correlation coefficient
[ ]( ) ( ) ( )xy E x t y t =
( )( ) = = ( ) ( ) ( ) ( )xy x y xy x yC E x t y t
0 0
= =
( ) ( )( )( )
( ) ( )
y xyxxy
x y xx yy
y t Cx tr E
C C
-
8/10/2019 Correlation Functions in Time and Frequency Domain
9/48
9SIPE, lecture 2 | xx
Example: Estimation of a Time Delay
Consider the system:
Additive noise v(t) is stochastic and uncorrelated to x(t),
Cross-covariance of the measured input and output in case:
input x(t) is white noise (stochastic)
0( ) ( ) ( )y t x t v t = +
-
8/10/2019 Correlation Functions in Time and Frequency Domain
10/48
10SIPE, lecture 2 | xx
Example: Estimation of a Time Delay
Matlab
demo: Lec2_example_Cuy_delay.m
0 5 10 15 20 25 30 35 40 45 50
-5
0
5
u(t)
time [s]
u
0 5 10 15 20 25 30 35 40 45 50-5
0
5
y(t)
time [s]
u
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2-1
0
1
Cuy
()
[s]
-
8/10/2019 Correlation Functions in Time and Frequency Domain
11/48
11SIPE, lecture 2 | xx
Effects of Noise on Autocorrelations
Often, correlation functions must be estimated from measurements.
Consider x (t ) and y (t ), corrupted with noise n (t ) and v (t ) resp.:
Noises have zero mean and are independent of the signals and of eachother.
Autocorrelation:
Additive noise will bias autocorrelation functions!
( ) ( ) ( )
( ) ( ) ( )
w t x t n t
z t y t v t
= +
= +
( ) ( )( ) ( ) ( ) ( ) ( )zz E y t v t y t v t = + +
( ) ( ) ( ) ( )yy yv vy vv = + + +
( ) ( )yy vv = +
-
8/10/2019 Correlation Functions in Time and Frequency Domain
12/48
12SIPE, lecture 2 | xx
Effects of Noise on Cross-correlations
Cross-correlation:
Additive noise will not bias cross-correlation functions!
( )( )
= + +
= + + +
=
( ) ( ) ( ) ( ) ( )
( ) ( ) ( ) ( )
( )
wz
xy xv ny nv
xy
E x t n t y t v t
-
8/10/2019 Correlation Functions in Time and Frequency Domain
13/48
13SIPE, lecture 2 | xx
Noise Reduction
Longer recordings
More information, same amount of noise
Repeat the experiment Noise cancels out by averaging from exactly the same inputs
Improve Signal-to-Noise Ratio
Concentrate power at specified frequencies, assuming the noisepower remains the same
-
8/10/2019 Correlation Functions in Time and Frequency Domain
14/48
14SIPE, lecture 2 | xx
Properties of Estimators
Formula given (autocovariances
/ spectral densities) areestimators for the true
relations.
What are the properties of these estimators? bias / variance / consistency
Bias: structural error
Variance: random error
Consistent: A consistent estimator is an estimator that converges,in probability, to the quantity being estimated as the sample size
grows.
-
8/10/2019 Correlation Functions in Time and Frequency Domain
15/48
15SIPE, lecture 2 | xx
Example: estimator for mean value
and variance of a signal
Signal xk
, with
k=1N
Estimator for the signal mean:
Estimator for the signal variance:
What are the expected values of both estimators?
Expectation operator: E{.}
( )
=
=
=
=
N
k
xkx
N
k
kx
x
N
xN
1
22
1
1
1
-
8/10/2019 Correlation Functions in Time and Frequency Domain
16/48
16SIPE, lecture 2 | xx
Estimator for the mean value
{ } { }
{ } { }
{ } xN
k
xx
xk
N
k
k
N
k
kx
N
k
kx
NE
xExE
xENxNEE
xN
==
==
=
=
=
=
==
=
1
11
1
1
11
1
-
8/10/2019 Correlation Functions in Time and Frequency Domain
17/48
17SIPE, lecture 2 | xx
Estimator for the variance
Estimator is biased and consistent
( )
{ } { } { } { }
{ }
222
1 1 1
2
21 1 1 1
2 2
21 1 1 1
2 2 2 2 2 2 2
21
1 1 1
1 2 1
1 2 1
1 2 1 2
k l k l
N N N
x k x k l
k k l
N N N N
k k l l k k l l k
N N N N
x k k l l kk l l k
N
x x x x x x x x x x
k
x x xN N N
x x x x xN N N
E E x E x x E x xN N N
E K KN N N
= = =
= = = =
= = = =
=
= =
= +
= +
= + + +
{ }
1 1 1
2 2 2 2
22 1 1 1 1 1 1
N N N
k l l
x x x xNE N
N N N N
= = =
= + = =
-
8/10/2019 Correlation Functions in Time and Frequency Domain
18/48
18SIPE, lecture 2 | xx
Estimator for the variance
biased
estimator:
unbiased
estimator:
(default!)
( )
{ } 2221
22
111
1
xxx
N
k
xkx
N
N
NE
xN
=
=
= =
( )
{ } 221
22
1
1
xx
N
k
xkx
E
xN
=
= =
-
8/10/2019 Correlation Functions in Time and Frequency Domain
19/48
19SIPE, lecture 2 | xx
Estimates of Correlation Functions
Cross-correlation:
Let x(t) and y(t) are finite time realizations of ergodic
processes. Then:
Practically, signals are finite time and sampled every t.Signal x(t) is sampled at t=0, t, , (N-1)tgiving x(i) with i=1, 2, , N.
Then:
with i
=1, 2, , N
1
2 ( ) lim ( ) ( )
T
xyT
T
x t y t dtT
=
1
=
=
( ) ( ) ( )N
xy
i
x i y iN
[ ]( ) ( ) ( )xy E x t y t =
-
8/10/2019 Correlation Functions in Time and Frequency Domain
20/48
20SIPE, lecture 2 | xx
Estimates of Correlation Functions
Unbiased estimator:
Variance of the estimator increases with lag!To avoid this, divide by N (biased estimator):
Use large Nto minimize bias! I.e.
Similar estimators can be derived for the covariance andcorrelation coefficient functions
1
=
=
( ) ( ) ( )N
xyi
x i y iN
1 ( ) ( ) ( )N
xyi x i y iN =
=
1
N
N
-
8/10/2019 Correlation Functions in Time and Frequency Domain
21/48
21SIPE, lecture 2 | xx
Additional demos
Calculation of cross-covariance: Lec2_calculation_Cuy.m
1 ( ) ( ) ( )N
xy
i
x i y i
N
=
=
-
8/10/2019 Correlation Functions in Time and Frequency Domain
22/48
22SIPE, lecture 2 | xx
Cross-covariance of some basic systems
Example systems (Matlab
Demo: Lec2_example_systems.m):
Time delay
1st order
2nd order
Input signals:
White noise (WN)
Low pass filtered noise (LP: 1/(0.5s+1) )
ti d l 1st d t 2nd d t
-
8/10/2019 Correlation Functions in Time and Frequency Domain
23/48
23SIPE, lecture 2 | xx
-5 0 5-0.5
0
0.5
1time delay
Cuy
(WN)
-5 0 5-0.1
0
0.1
Cuy
(LP)
-5 0 50
0.5
1
impulse
time [s]
-5 0 5-0.01
0
0.01
0.021storder system
-5 0 5-0.02
0
0.02
0.04
-5 0 50
1
2
time [s]
-5 0 5-0.05
0
0.052ndorder system
-5 0 5-0.2
0
0.2
-5 0 5-5
0
5
10
time [s]
-
8/10/2019 Correlation Functions in Time and Frequency Domain
24/48
25SIPE, lecture 2 | xx
Intermezzo: Fourier Transform
Fourier Transform:
Mapping between time-domain and frequency-domain
One-to-one mapping
Unique: inverse Fourier Transform exists
Linear technique
Inverse Fourier Transform:
( )( ) ( ) 2( ) j ftX f x t x t e dt
= =
( )( ) ( )1 2( ) j tfx t X f X f e df
= =
-
8/10/2019 Correlation Functions in Time and Frequency Domain
25/48
26SIPE, lecture 2 | xx
Fourier transformation
y(t) is an arbitrary signal
Euler formula:
symmetric part: cos(z)
anti-symmetric part: sin(z)
2( ) ( ) j ftY f y t e dt =
2 2 2( ) ( ) cos(2 ) *sin(2 )j ft j ft j fte re e im e ft j ft = + =
( ) ( )cos sinjze z j z = +
-
8/10/2019 Correlation Functions in Time and Frequency Domain
26/48
27SIPE, lecture 2 | xx
Example Fourier Transformation
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-1.5
-1
-0.5
0
0.5
1
1.5
( ) ( )HzHzty 5sin5.0sin)( +=
Y(0.5Hz): 5 Hz signal will be averaged out
2( ) ( ) j ftY f y t e dt =
-
8/10/2019 Correlation Functions in Time and Frequency Domain
27/48
28SIPE, lecture 2 | xx
Example Fourier Transformation
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-1.5
-1
-0.5
0
0.5
1
1.5
-0.5 0 0.5 1 1.50
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
y(t) = sin(0.5t) Y() = (-0.5)
-
8/10/2019 Correlation Functions in Time and Frequency Domain
28/48
29SIPE, lecture 2 | xx
Frequency Domain Expressions
Discrete Fourier Transform:
where f
takes values 0, 1, , N-1
multiples of
Inverse Fourier Transform:
( )2
1
( ) ( ) ( )ftN jN
t
U f u t u t e
=
= =
( )2
1
1
1( ) ( ) ( )
ftN jN
f
u t U f U f eN
=
= =
1f
N t =
-
8/10/2019 Correlation Functions in Time and Frequency Domain
29/48
30SIPE, lecture 2 | xx
Fourier transform of signals
Discrete Fourier Transform (DFT)
DFT maps N real values in time domain to N complex values infrequency domain
Double information? DFT is symmetric U(-r)=U(r)* Information for N/2 complex values
( ) [ ]
( ){ } ( )
[ ]1,,1,0);(
)(
1,,1,0;
1
0
/2
==
=
NrrU
ekukuDFTrU
Nkku
N
k
Nrkj
2,,1,0
Nr
P t
-
8/10/2019 Correlation Functions in Time and Frequency Domain
30/48
31SIPE, lecture 2 | xx
Power spectrum or
auto-spectral density
auto-spectral density Sxx
is Fourier Transform of autocorrelation
properties:
Real values (no imaginary part)
Symmetry: Sxx
(f)=Sxx
(-f) => only positive frequencies are analyzed
Cxx
(0) =
Sxx
(f) df
= x2
Area under Sxx
(f) is equal to variance of signal (Parsevals
theorem)
( )
( ) 2
( ) ( ) ( )
( ) ( ) ( )
j
xx xx xx
jf
xx xx xx
S e d
S f e d
= =
= =
C t
-
8/10/2019 Correlation Functions in Time and Frequency Domain
31/48
32SIPE, lecture 2 | xx
Cross-spectrum orCross-spectral density
cross-spectral density Sxy
:
properties:
Complex values Sxy
*(f)=Sxy
(-f) => only positive frequencies are analyzed
Sxy
(f) describes the interdependency of signals x(t) and y(t) in
frequency domain (gain and phase)
if xy
() = 0, then Sxy
(f) = 0 for all frequencies
( )
2( ) ( ) ( ) j fxy xy xy
S f e d
= =
-
8/10/2019 Correlation Functions in Time and Frequency Domain
32/48
33SIPE, lecture 2 | xx
Estimators for the power spectrum
Fourier transform of the autocorrelation function, the power spectrum:
using the time average estimate:
and multiplication with
gives:
* : complex conjugate:
1 2
0
( ) ( )fN jN
uu uuS f e
=
=
1 2 2
0
1 ( )
( ) ( ) ( )i f if N Nj j
N Nuu
i
S f u i e u i eN
= =
=
1 ( ) ( ) ( )N
uui
u i u iN
=
=
20 1
= =( )i i f
jNe e
1 *( ) ( )U n f U n f N
= *( ) ( )U n f U n f =
Indirect approach
and direct approachgive equal result!
-
8/10/2019 Correlation Functions in Time and Frequency Domain
33/48
34SIPE, lecture 2 | xx
Estimator for the cross-spectrum
Fourier transform of the cross-correlation function is called the cross-spectrum:
1 * ( ) ( ) ( )uy
S f U n f Y n f N
=
1 2
0
==
( ) ( )
fN jN
uy uyS f e
-
8/10/2019 Correlation Functions in Time and Frequency Domain
34/48
35SIPE, lecture 2 | xx
Time-domain vs. Frequency-domain
x(t), y(t)
xy
()
Cxy
()
rxy
()
X(f), Y(f)
Sxy
(f)
xy
(f)
input, output
cross-correlation
function
cross-covariance
function
correlation
coefficient
input, output
cross-spectral
density
coherence
FourierTransformationTime
Domain Frequency Domain
-
8/10/2019 Correlation Functions in Time and Frequency Domain
35/48
36SIPE, lecture 2 | xx
Discrete and Continuous Signals
Reconstruction
DA conversion
Distortion: Hold Circuits
Sampling
AD conversion
Distortion: "Dirac Comb"
Continuous system: Plant
Digital Controller: Plant performance enhancement
-
8/10/2019 Correlation Functions in Time and Frequency Domain
36/48
37SIPE, lecture 2 | xx
Sampling, Diracs Comb
-
8/10/2019 Correlation Functions in Time and Frequency Domain
37/48
38SIPE, lecture 2 | xx
Models of Linear Systems
System:
Linear systems obey both scaling and superposition property!
( ) ( ( ))y t N u t=
1 1 1 1
2 2 2 2
1 1 2 2 1 1 2 2
=
=
+ = +
( ) ( ( ))
( ) ( ( ))
( ) ( ) ( ( ) ( ))
k y t N k u t
k y t N k u t
k y t k y t N k u t k u t
N
u(t) y(t)
-
8/10/2019 Correlation Functions in Time and Frequency Domain
38/48
39SIPE, lecture 2 | xx
Time domain models
Assume input is a pulse with:
Response of linear system N to
pulse:
Assume u(t) is weighted sum of
pulses:
( )1
for 0, 2
0 otherwise
tt
d t t t
-
8/10/2019 Correlation Functions in Time and Frequency Domain
39/48
40SIPE, lecture 2 | xx
Time domain models
The output is:
Limit case: unit-area pulse d(t,t) is an impulse:
Then h(t) is the impulse response function (IRF) and the output
the convolution integral
( ) ( )0
lim ,t
d t t t
=
( )( )
( )
=
=
=
=
=
( ) ( ( ))
,
,
kk
k
k
y t N u t
u N d t k t t
u h t k t t
( ) ( )
= ( )y t h u t d
-
8/10/2019 Correlation Functions in Time and Frequency Domain
40/48
41SIPE, lecture 2 | xx
Time domain models
Causal system: h(t)=0 for tT
Discrete
( ) ( )
1
0
== ( )
T
y t h u t
( ) ( ) = ( )T
o
y t h u t d
-
8/10/2019 Correlation Functions in Time and Frequency Domain
41/48
42SIPE, lecture 2 | xx
Frequency domain models
Time-domain: convolution integral
Fourier transform:( ) ( ) ( )
( ) ( )
( ) ( )
( )
2
2
2
= =
=
=
=
=
( ) ( )
( )
( ) ( ) ( )
j ft
ft
f
Y f y t h u t d
h u t d e dt
h u t e dtd
U f h e d
Y f H f U f
( ) ( )
= ( )y t h u t d
Convolution in time domain is multiplication infrequency domain
(and vice-versa)
-
8/10/2019 Correlation Functions in Time and Frequency Domain
42/48
43SIPE, lecture 2 | xx
Estimators in time domain
Cross-covariance
When analyzing the dynamics of a dynamical system interested in
variations of the signals around it mean (and not average value).
Assuming signals with zero-mean
( ) ( ){ }
( ) ( )( ) ( ') ( ') '
( ) ( ') ( ) ( ') 'uy
y t h u t h t u t t dt
C E u t y t E h t u t u t t dt
= =
= =
Basic identification with cross-
-
8/10/2019 Correlation Functions in Time and Frequency Domain
43/48
44SIPE, lecture 2 | xx
Basic identification with cross-covariance
( ) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( ) ( )
( ) ( ) ( ) ( )
( ) ( )
'
multiply with ( ): ' ( )
' '
white noise: 0 for 0; 0 1
uy un uu
uu uu
y t n t h t u t - t' dt'
u t u t y t u t n t h t u t u t - t' dt'
C C h t' C t dt
C C
= +
= +
= +
= =
( ) ( ) ( )
Other 'tricks' needed when u(t) is not white
uy unC C h = +
?y(t)
u(t)
n(t)
Basic identification with spectral
-
8/10/2019 Correlation Functions in Time and Frequency Domain
44/48
45SIPE, lecture 2 | xx
Basic identification with spectral
densities
( ) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( ) ( )
( ) ( ) ( ) ( )
( ) ( ) ( ) ( )
( ) ( )
'
multiply with ( ): ' ( )
' '
Fourier transform:
if 0:
uy un uu
uy un uu
un
y t n t h t u t - t' dt'
u t u t y t u t n t h t u t u t - t' dt'
C C h t' C t dt
S f S f H f S f
SS f H f
= +
= +
= +
= +
= =
( )
( )uy
uu
f
S f
?y(t)
u(t)
n(t)
Identification:
-
8/10/2019 Correlation Functions in Time and Frequency Domain
45/48
46SIPE, lecture 2 | xx
Identification:
time-domain vs. frequency-domain
Time domain
Unknown system: impulse response function (IRF) of h(t) over
limited time Mostly: direct model parameterization (fit of physics model)
Frequency domain
Unknown system: frequency response function (FRF) H(f) for number
of frequencies
?y(t)
u(t)
n(t)
Example IRF & FRF of typical
-
8/10/2019 Correlation Functions in Time and Frequency Domain
46/48
47SIPE, lecture 2 | xx
Example IRF & FRF of typical
systems
Time delay
First order system
Second order system
Unstable system
(try Matlab!)
-
8/10/2019 Correlation Functions in Time and Frequency Domain
47/48
48SIPE, lecture 2 | xx
Book: Westwick
& Kearney
Lecture 1: signals
Chapter 1, all
Chapter 2, sec. 2.1
2.3.1
Lecture 2: Correlation functions in time & frequency domain
Chapter 2, sec. 2.3.2
2.3.4
Chapter 3, sec. 3.1
3.2
Lecture 3: Estimators for impulse & frequency response functions
Chapter 5, sec. 5.1
5.3
-
8/10/2019 Correlation Functions in Time and Frequency Domain
48/48
49SIPE, lecture 2 | xx
Next week: lecture 3
Lecture 3:
Estimation of IRF and FRF
Estimator for coherence
Open loop and closed loop system identification