Time series analysis Matlab tutorial

Joachim Gross

Outline• Terminology• Sampling theorem• Plotting• Baseline correction• Detrending• Smoothing• Filtering• Decimation

Remarks• Focus on practical aspects, exercises, getting

experience (not on equations, theory)• Focus on “How to do …”• Learn some basic skills for TS analysis

• Note: Usually there is not a single perfectly correct way of doing a TS operation! => learn the limitations!

What is a time series?A sequence of measurements over time

0 2 4 6 8 10-1

-0.5

0

0.5

1

1.5

Terminology– Continuous TS: continuous observations– Discrete TS: observations at specific times usually equally spaced

– Deterministic TS: future values can be exactly predicted from past values

– Stochastic TS: exact prediction not possible

Objectives of TS analysis• Description• Explanation• Prediction• Control

Simple descriptive analysisSummary statistics (mean, std) is not always meaningful for

TS

0 50 100 150 200 250 300 350 400 450 5000

2

4

6

8

10

12

Sampling• Converting a continuous signal into a discrete time series• Reconstruction is possible if sampling frequency is greater than twice

the signal bandwidth

0 0.2 0.4 0.6 0.8 1-1

-0.5

0

0.5

1

Time (s)0 0.2 0.4 0.6 0.8 1

-1

-0.5

0

0.5

1

75 Hz sampling

Sampling• Nyquist frequency: half of sampling frequency

10 Hz sampling 10 Hz reconstruction

0 0.2 0.4 0.6 0.8 1-3

-2

-1

0

1

2

3

0 0.2 0.4 0.6 0.8 1-1

-0.5

0

0.5

1

Sampling

8 Hz sampling

0 0.2 0.4 0.6 0.8 1-1

-0.5

0

0.5

1

• Aliasing: Frequencies above Nyquist frequency are reconstructed below Nyquist frequency

Sampling

8 Hz sampling

• Aliasing: Frequencies above Nyquist frequency are reconstructed below Nyquist frequency

0 1 2 3 4-140

-120

-100

-80

-60

-40

-20

0

20

Frequency

Pow

er S

pect

rum

Mag

nitu

de (d

B)

0 5 10 15 20-140

-120

-100

-80

-60

-40

-20

0

20

Frequency

Pow

er S

pect

rum

Mag

nitu

de (d

B)

40 Hz sampling

Simple operations on TS• Plotting• Removing a baseline• Removing a trend• Smoothing• Filtering• Decimation

Plotting in Matlab• For visual inspection of TS• For publications/talks

• plot• sptool

Data preprocessing I• Removing offset• ts=ts-mean(ts);

0 0.2 0.4 0.6 0.8 10

2

4

6

8

10

0 0.2 0.4 0.6 0.8 1-1

-0.5

0

0.5

1

Data preprocessing I• Removing a baseline• basel=find(t<=0);• ts=ts-mean(ts(basel));

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6-2

-1

0

1

2

3

4

5

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6-3

-2

-1

0

1

2

3

4

Data preprocessing II• Removing a trend• ts=detrend(ts);• subtracts best fitting line• detrend can be used to subtract mean: detrend(ts,’constant’)

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6-2

-1

0

1

2

3

4

5

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6-3

-2

-1

0

1

2

3

4

5

data 1 lineardata 2

Data preprocessing III• Smoothing• ts=filter(ones(1,30)/30,1,ts); %mean filter, moving average• uses zeros at beginning! • => baseline correction or do not use first 30 samples

-0.2 0 0.2 0.4 0.60

1

2

3

4

5

6

7

Data preprocessing III• introduces a shift! => either correct for it or • ts=filtfilt(ones(1,15)/15,1,ts); %mean filter, forward and reverse• no shift!• filter can take any smoothing kernel (gaussian, etc)

-0.2 0 0.2 0.4 0.6-3

-2

-1

0

1

2

3

4

-0.2 0 0.2 0.4 0.60

1

2

3

4

5

6

7shifted by 15 samples filtfilt

Data preprocessing III• Smoothing• ts=medfilt1(ts,30); %median filter, takes into account the shift• uses 0 at beginning and end !

-0.2 -0.18 -0.16 -0.14 -0.12 -0.11

1.5

2

2.5

3

3.5

4

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.60

1

2

3

4

5

6

7

Data preprocessing III• Smoothing• ts=sgolayfilt(ts,3,41); %Savitzky-Golay filter• fits 3rd order polynomial to frames of size 41• good at preserving high frequencies in the data

-0.2 0 0.2 0.4 0.60

1

2

3

4

5

6

7

Data preprocessing III• Smoothing• compare unsmoothed and smoothed data• check for shift• check beginning (and end) of the smoothed time series

Exercise 1

Data preprocessing IV• Filtering• FIR-Filter (finite impulse

response)• stable• high filter order• usually have linear phase(phase change is proportional to

frequency)

• IIR-Filter (infinite impulse response)

• potentially unstable• low filter order• non-linear phase distortion• computationally efficient

Data preprocessing IV• IIR-Filter:

– Butterworth– Elliptic– Chebychev Typ 1– Chebychev Typ 2– Bessel

• FIR-Filter:– fir1

Data preprocessing IV• lowpass• highpass• bandpass• bandstop

0 5 10 15 20 25 30 35 40 45

-400

-350

-300

-250

-200

-150

-100

-50

0

Frequency (Hz)

Mag

nitu

de (d

B)

Magnitude Response (dB)

dB is logarithmic unit0dB = factor of 13dB = factor of 210dB= factor of 10

5 Hz lowpass

Data preprocessing IV• lowpass• highpass• bandpass• bandstop

dB is logarithmic unit0dB = factor of 13dB = factor of 210dB= factor of 10

0 5 10 15 20 25 30 35 40 45

-100

-90

-80

-70

-60

-50

-40

-30

-20

-10

0

Frequency (Hz)

Mag

nitu

de (d

B)

Magnitude Response (dB)

30 Hz highpass

Data preprocessing IV• lowpass• highpass• bandpass• bandstop

dB is logarithmic unit0dB = factor of 13dB = factor of 210dB= factor of 10

2-30 Hz bandpass

0 5 10 15 20 25 30 35 40 45

-40

-35

-30

-25

-20

-15

-10

-5

0

Frequency (Hz)

Mag

nitu

de (d

B)

Magnitude Response (dB)

Data preprocessing IV• lowpass• highpass• bandpass• bandstop

dB is logarithmic unit0dB = factor of 13dB = factor of 210dB= factor of 10

30-40 Hz bandstop

0 5 10 15 20 25 30 35 40 45

-25

-20

-15

-10

-5

0

Frequency (Hz)

Mag

nitu

de (d

B)

Magnitude Response (dB)

Simple design: FIR• [b]=fir1(4,2*4/sf); %4 Hz lowpass• [b]=fir1(4,2*4/sf,’high’); %4 Hz highpass• [b]=fir1(4,2*[4 10]/sf); %4-10 Hz bandpass• [b]=fir1(4,2*[4 10]/sf,’stop’); %4-10 Hz bandstop

• tsf=filter(b,1,ts);• tsf=filtfilt(b,1,ts); %forward and reverse

Simple design: IIR• [b,a]=butter(4,2*4/sf); %4 Hz lowpass• [b,a]=butter(4,2*4/sf,’high’); %4 Hz highpass• [b,a]=butter(4,2*[4 10]/sf); %4-10 Hz bandpass• [b,a]=butter(4,2*[4 10]/sf,’stop’); %4-10 Hz bandstop

• tsf=filter(b,a,ts);• tsf=filtfilt(b,a,ts); %forward and reverse

Simple Inspection

0 5 10 15 20 25 30 35 40 45 50-400

-300

-200

-100

0

Frequency (Hz)

Pha

se (d

egre

es)

0 5 10 15 20 25 30 35 40 45 50-300

-200

-100

0

Frequency (Hz)

Mag

nitu

de (d

B)

freqz(b,a,100,100);sf

number of frequencies

Complex design• fdatool

– magnitude response– phase response– impulse response– compare filters– effect of changing filter order

Filter artifacts• onset transients

0 2 4 6 8-6

-5

-4

-3

-2

-1

0

1

2x 10-3 1 Hz lowpass (4th order Butterworth)

0 2 4 6 8-6

-5

-4

-3

-2

-1

0

1

2x 10-3

filtfiltfilter

Filter artifacts• ringing

with artifact

3.5 4 4.5-5

0

5

10

15

20x 10-3

3.5 4 4.5-5

0

5

10

15

20x 10-3

without artifact

Filter artifacts• ringing

filter, 20 Hz lowpass (12th order Butterworth)

3.5 4 4.5-5

0

5

10

15

20x 10-3

3.5 4 4.5

-2

-1

0

1

2

3

4x 10-3

filtfilt

Filter artifacts• beginning and end of filtered ts is distorted• filtering artifacts is dangerous• filtering may change the latency of effects!• filtering may change the phase

Suggestions• be careful with low frequencies• use low order butterworth forward and reverse (to avoid

phase distortions)• carefully check beginning and end of filtered ts• make sure you don’t have artifacts in the data• use surrogate data (filtered noise)

Data preprocessing V• Decimation• ts=decimate(ts,4);• decimate uses a lowpass filter to avoid aliasing artifacts

Exercises 2-4