Introduction to Random Processes - University of Edinburgh ?· Introduction to Random Processes UDRC…

Download Introduction to Random Processes - University of Edinburgh ?· Introduction to Random Processes UDRC…

Post on 11-Jul-2018

212 views

Category:

Documents

0 download

TRANSCRIPT

- p. 1/55THEUN IV ERSITYOFEDI N BURGHIntroduction to Random ProcessesUDRC Summer School, 27th June 2016Dr James R. HopgoodJames.Hopgood@ed.ac.ukRoom 2.05Alexander Graham Bell BuildingThe Kings BuildingsInstitute for Digital CommunicationsSchool of EngineeringCollege of Science and EngineeringUniversity of EdinburghStochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 2/55THEUN IV ERSITYOFEDI N BURGHBlank PageThis slide is intentionally left blank.- p. 3/55Handout 2Stochastic ProcessesStochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription usingprobability densityfunctions (pdfs)Second-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 4/55THEUN IV ERSITYOFEDI N BURGHDefinition of a Stochastic ProcessNatural discrete-time signals can be characterised as randomsignals, since their values cannot be determined precisely;they are unpredictable.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 4/55THEUN IV ERSITYOFEDI N BURGHDefinition of a Stochastic ProcessNatural discrete-time signals can be characterised as randomsignals, since their values cannot be determined precisely;they are unpredictable.Consider an experiment with outcomes S = {k, k Z+},each occurring with probability Pr (k). Assign to each k Sa deterministic sequence x[n, k] , n Z.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 4/55THEUN IV ERSITYOFEDI N BURGHDefinition of a Stochastic ProcessNatural discrete-time signals can be characterised as randomsignals, since their values cannot be determined precisely;they are unpredictable.Consider an experiment with outcomes S = {k, k Z+},each occurring with probability Pr (k). Assign to each k Sa deterministic sequence x[n, k] , n Z.The sample space S, probabilities Pr (k), and the sequencesx[n, k] , n Z constitute a discrete-time stochastic process,or random sequence.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 4/55THEUN IV ERSITYOFEDI N BURGHDefinition of a Stochastic ProcessNatural discrete-time signals can be characterised as randomsignals, since their values cannot be determined precisely;they are unpredictable.Consider an experiment with outcomes S = {k, k Z+},each occurring with probability Pr (k). Assign to each k Sa deterministic sequence x[n, k] , n Z.The sample space S, probabilities Pr (k), and the sequencesx[n, k] , n Z constitute a discrete-time stochastic process,or random sequence.Formally, x[n, k] , n Z is a random sequence or stochasticprocess if, for a fixed value n0 Z+ of n, x[n0, ] , n Z is arandom variable.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 4/55THEUN IV ERSITYOFEDI N BURGHDefinition of a Stochastic ProcessNatural discrete-time signals can be characterised as randomsignals, since their values cannot be determined precisely;they are unpredictable.Consider an experiment with outcomes S = {k, k Z+},each occurring with probability Pr (k). Assign to each k Sa deterministic sequence x[n, k] , n Z.The sample space S, probabilities Pr (k), and the sequencesx[n, k] , n Z constitute a discrete-time stochastic process,or random sequence.Formally, x[n, k] , n Z is a random sequence or stochasticprocess if, for a fixed value n0 Z+ of n, x[n0, ] , n Z is arandom variable.Also known as a time series in the statistics literature.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 4/55THEUN IV ERSITYOFEDI N BURGHDefinition of a Stochastic ProcessNatural discrete-time signals can be characterised as randomsignals, since their values cannot be determined precisely;they are unpredictable.Consider an experiment with outcomes S = {k, k Z+},each occurring with probability Pr (k). Assign to each k Sa deterministic sequence x[n, k] , n Z.The sample space S, probabilities Pr (k), and the sequencesx[n, k] , n Z constitute a discrete-time stochastic process,or random sequence.Formally, x[n, k] , n Z is a random sequence or stochasticprocess if, for a fixed value n0 Z+ of n, x[n0, ] , n Z is arandom variable.Also known as a time series in the statistics literature.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 5/55THEUN IV ERSITYOFEDI N BURGHInterpretation of SequencesA graphical representation of a random process.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 5/55THEUN IV ERSITYOFEDI N BURGHInterpretation of SequencesThe set of all possible sequences {x[n, ]} is called an ensemble,and each individual sequence x[n, k], corresponding to aspecific value of = k, is called a realisation or a samplesequence of the ensemble.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 5/55THEUN IV ERSITYOFEDI N BURGHInterpretation of SequencesThe set of all possible sequences {x[n, ]} is called an ensemble,and each individual sequence x[n, k], corresponding to aspecific value of = k, is called a realisation or a samplesequence of the ensemble.There are four possible interpretations of x[n, ]: Fixed Variablen Fixed Number Random variablen Variable Sample sequence Stochastic processStochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 5/55THEUN IV ERSITYOFEDI N BURGHInterpretation of SequencesThe set of all possible sequences {x[n, ]} is called an ensemble,and each individual sequence x[n, k], corresponding to aspecific value of = k, is called a realisation or a samplesequence of the ensemble.There are four possible interpretations of x[n, ]: Fixed Variablen Fixed Number Random variablen Variable Sample sequence Stochastic processUse simplified notation x[n] x[n, ] to denote both a stochasticprocess, and a single realisation. Use the terms random processand stochastic process interchangeably throughout this course.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 6/55THEUN IV ERSITYOFEDI N BURGHPredictable ProcessesThe unpredictability of a random process is, in general, thecombined result of the following two characteristics:1. The selection of a single realisation is based on the outcome ofa random experiment;2. No functional description is available for all realisations of theensemble.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 6/55THEUN IV ERSITYOFEDI N BURGHPredictable ProcessesThe unpredictability of a random process is, in general, thecombined result of the following two characteristics:1. The selection of a single realisation is based on the outcome ofa random experiment;2. No functional description is available for all realisations of theensemble.In some special cases, however, a functional relationship isavailable. This means that after the occurrence of all samples ofa particular realisation up to a particular point, n, all futurevalues can be predicted exactly from the past ones.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 6/55THEUN IV ERSITYOFEDI N BURGHPredictable ProcessesThe unpredictability of a random process is, in general, thecombined result of the following two characteristics:1. The selection of a single realisation is based on the outcome ofa random experiment;2. No functional description is available for all realisations of theensemble.In some special cases, however, a functional relationship isavailable. This means that after the occurrence of all samples ofa particular realisation up to a particular point, n, all futurevalues can be predicted exactly from the past ones.If this is the case for a random process, then it is calledpredictable, otherwise it is said to be unpredictable or aregular process.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 7/55THEUN IV ERSITYOFEDI N BURGHDescription using pdfsFor fixed n = n0, x[n0, ] is a random variable. Moreover, therandom vector formed from the k random variables{x[nj ] , j {1, . . . k}} is characterised by the cumulativedistribution function (cdf) and pdfs:FX (x1 . . . xk | n1 . . . nk) = Pr (x[n1] x1, . . . , x[nk] xk)fX (x1 . . . xk | n1 . . . nk) =kFX (x1 . . . xk | n1 . . . nk)x1 xkStochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 7/55THEUN IV ERSITYOFEDI N BURGHDescription using pdfsFor fixed n = n0, x[n0, ] is a random variable. Moreover, therandom vector formed from the k random variables{x[nj ] , j {1, . . . k}} is characterised by the cdf and pdfs:FX (x1 . . . xk | n1 . . . nk) = Pr (x[n1] x1, . . . , x[nk] xk)fX (x1 . . . xk | n1 . . . nk) =kFX (x1 . . . xk | n1 . . . nk)x1 xkIn exactly the same way as with random variables and randomvectors, it is:difficult to estimate these probability functions withoutconsiderable additional information or assumptions;possible to frequently characterise stochastic processesusefully with much less information.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 8/55THEUN IV ERSITYOFEDI N BURGHSecond-order Statistical DescriptionMean and Variance Sequence At time n, the ensemble mean andvariance are given by:x[n] = E [x[n]]2x[n] = E[|x[n] x[n] |2]= E[|x[n] |2] |x[n] |2Both x[n] and 2x[n] are deterministic sequences.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 8/55THEUN IV ERSITYOFEDI N BURGHSecond-order Statistical DescriptionMean and Variance Sequence At time n, the ensemble mean andvariance are given by:x[n] = E [x[n]]2x[n] = E[|x[n] x[n] |2]= E[|x[n] |2] |x[n] |2Both x[n] and 2x[n] are deterministic sequences.Autocorrelation sequence The second-order statistic rxx[n1, n2]provides a measure of the dependence between values of theprocess at two different times; it can provide informationabout the time variation of the process:rxx[n1, n2] = E [x[n1] x[n2]]Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 8/55THEUN IV ERSITYOFEDI N BURGHSecond-order Statistical DescriptionAutocovariance sequence The autocovariance sequence provides ameasure of how similar the deviation from the mean of aprocess is at two different time instances:xx[n1, n2] = E [(x[n1] x[n1])(x[n2] x[n2])]= rxx[n1, n2] x[n1] x[n2]Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 8/55THEUN IV ERSITYOFEDI N BURGHSecond-order Statistical DescriptionAutocovariance sequence The autocovariance sequence provides ameasure of how similar the deviation from the mean of aprocess is at two different time instances:xx[n1, n2] = E [(x[n1] x[n1])(x[n2] x[n2])]= rxx[n1, n2] x[n1] x[n2]To show how these deterministic sequences of a stochasticprocess can be calculated, several examples are considered indetail below.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 9/55THEUN IV ERSITYOFEDI N BURGHExample of calculating autocorrelationsExample ( [Manolakis:2000, Ex 3.9, page 144]). The harmonic processx[n] is defined by:x[n] =Mk=1Ak cos(kn+ k), k 6= 0where M , {Ak}M1 and {k}M1 are constants, and {k}M1 arepairwise independent random variables uniformly distributed inthe interval [0, 2].1. Determine the mean of x(n).2. Show the autocorrelation sequence is given byrxx[] =12Mk=1|Ak|2 cosk, < < Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 9/55THEUN IV ERSITYOFEDI N BURGHExample of calculating autocorrelationsExample ( [Manolakis:2000, Ex 3.9, page 144]). SOLUTION. 1. Theexpected value of the process is straightforwardly given by:E [x(n)] = E[Mk=1Ak cos(kn+ k)]=Mk=1Ak E [cos(kn+ k)]Since a co-sinusoid is zero-mean, then:E [cos(kn+ k)] = 20cos(kn+ k)12 dk = 0Hence, it follows:E [x(n)] = 0, n Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 9/55THEUN IV ERSITYOFEDI N BURGHExample of calculating autocorrelationsExample ( [Manolakis:2000, Ex 3.9, page 144]). SOLUTION. 1.rxx(n1, n2) = EMk=1Ak cos(kn1 + k)Mj=1Aj cos(jn2 + j)=Mk=1Mj=1Ak AjE [cos(kn1 + k) cos(jn2 + j)]After some algebra, it can be shown that:E [cos(kn1 + k) cos(jn2 + j)] ={12 cosk(n1 n2) k = j0 otherwisewhere g(k) = cos(kn1 + k) and h(k) = cos(jn2 + j),and the fact that k and j are independent implies theexpectation function may be factorised.E [cos( n + ) cos( n + )] =1cos (n n ) (k j)Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 10/55THEUN IV ERSITYOFEDI N BURGHTypes of Stochastic ProcessesIndependence A stochastic process is independent if, and onlyif, (iff)fX (x1, . . . , xN | n1, . . . , nN ) =Nk=1fXk (xk | nk)N, nk, k {1, . . . , N}. Here, therefore, x(n) is a sequenceof independent random variables.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 10/55THEUN IV ERSITYOFEDI N BURGHTypes of Stochastic ProcessesIndependence A stochastic process is independent ifffX (x1, . . . , xN | n1, . . . , nN ) =Nk=1fXk (xk | nk)N, nk, k {1, . . . , N}. Here, therefore, x(n) is a sequenceof independent random variables.An independent and identically distributed (i. i. d.) process is onewhere all the random variables {x(nk, ), nk Z} have thesame pdf, and x(n) will be called an i. i. d. random process.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 10/55THEUN IV ERSITYOFEDI N BURGHTypes of Stochastic ProcessesIndependence A stochastic process is independent ifffX (x1, . . . , xN | n1, . . . , nN ) =Nk=1fXk (xk | nk)N, nk, k {1, . . . , N}. Here, therefore, x(n) is a sequenceof independent random variables.An i. i. d. process is one where all the random variables{x(nk, ), nk Z} have the same pdf, and x(n) will be calledan i. i. d. random process.An uncorrelated processes is a sequence of uncorrelated randomvariables:xx(n1, n2) = 2x(n1) (n1 n2)Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 10/55THEUN IV ERSITYOFEDI N BURGHTypes of Stochastic ProcessesAn orthogonal process is a sequence of orthogonal randomvariables, and is given by:rxx(n1, n2) = E[|x(n1)|2](n1 n2)If a process is zero-mean, then it is both orthogonal anduncorrelated since xx(n1, n2) = rxx(n1, n2).Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 10/55THEUN IV ERSITYOFEDI N BURGHTypes of Stochastic ProcessesAn orthogonal process is a sequence of orthogonal randomvariables, and is given by:rxx(n1, n2) = E[|x(n1)|2](n1 n2)If a process is zero-mean, then it is both orthogonal anduncorrelated since xx(n1, n2) = rxx(n1, n2).A stationary process is a random process where its statisticalproperties do not vary with time. Processes whose statisticalproperties do change with time are referred to asnonstationary.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 11/55THEUN IV ERSITYOFEDI N BURGHStationary ProcessesA random process x(n) has been called stationary if its statisticsdetermined for x(n) are equal to those for x(n+ k), for every k.There are various formal definitions of stationarity, along withquasi-stationary processes, which are discussed below.Order-N and strict-sense stationarityWide-sense stationarityWide-sense periodicity and cyclo-stationarityLocal- or quasi-stationary processesAfter this, some examples of various stationary processes will begiven.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 12/55THEUN IV ERSITYOFEDI N BURGHOrder-N and strict-sense stationarityDefinition (Stationary of order-N ). A stochastic process x(n) iscalled stationary of order-N if:fX (x1, . . . , xN | n1, . . . , nN ) = fX (x1, . . . , xN | n1 + k, . . . , nN + k)for any value of k. If x(n) is stationary for all orders N Z+, it issaid to be strict-sense stationary (SSS).Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 12/55THEUN IV ERSITYOFEDI N BURGHOrder-N and strict-sense stationarityDefinition (Stationary of order-N ). A stochastic process x(n) iscalled stationary of order-N if:fX (x1, . . . , xN | n1, . . . , nN ) = fX (x1, . . . , xN | n1 + k, . . . , nN + k)for any value of k. If x(n) is stationary for all orders N Z+, it issaid to be SSS.An independent and identically distributed process is SSS since,in this case, fXk (xk | nk) = fX (xk) is independent of n, andtherefore also of n+ k.However, SSS is more restrictive than necessary in practicalapplications, and is a rarely required property.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 13/55THEUN IV ERSITYOFEDI N BURGHWide-sense stationarityA more relaxed form of stationarity, which is sufficient forpractical problems, occurs when a random process is stationaryorder-2; such a process is wide-sense stationary (WSS).Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 13/55THEUN IV ERSITYOFEDI N BURGHWide-sense stationarityDefinition (Wide-sense stationarity). A random signal x(n) is calledwide-sense stationary if:the mean and variance is constant and independent of n:E [x(n)] = xvar [x(n)] = 2xthe autocorrelation depends only on the time differencel = n1 n2, called the lag:rxx(n1, n2) = rxx(n2, n1) = E [x(n1)x(n2)]= rxx(l) = rxx(n1 n2) = E [x(n1)x(n1 l)]= E [x(n2 + l)x(n2)]Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 13/55THEUN IV ERSITYOFEDI N BURGHWide-sense stationarityThe autocovariance function is given by:xx(l) = rxx(l) |x|2Since 2nd-order moments are defined in terms of 2nd-orderpdf, then strict-sense stationary are always WSS, but notnecessarily vice-versa, except if the signal is Gaussian.In practice, however, it is very rare to encounter a signal thatis stationary in the wide-sense, but not stationary in the strictsense.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 14/55THEUN IV ERSITYOFEDI N BURGHWide-sense cyclo-stationarityTwo classes of nonstationary process which, in part, haveproperties resembling stationary signals are:1. A wide-sense periodic (WSP) process is classified as signals whosemean is periodic, and whose autocorrelation function isperiodic in both dimensions:x(n) = x(n+N)rxx(n1, n2) = rxx(n1 +N,n2) = rxx(n1, n2 +N)= rxx(n1 +N,n2 +N)for all n, n1 and n2. These are quite tight constraints for realsignals.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 14/55THEUN IV ERSITYOFEDI N BURGHWide-sense cyclo-stationarity2. A wide-sense cyclo-stationary process has similar but lessrestrictive properties than a WSP process, in that the mean isperiodic, but the autocorrelation function is now justinvariant to a shift by N in both of its arguments:x(n) = x(n+N)rxx(n1, n2) = rxx(n1 +N,n2 +N)for all n, n1 and n2. This type of nonstationary process hasmore practical applications, as the following example willshow.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 15/55THEUN IV ERSITYOFEDI N BURGHQuasi-stationarityAt the introduction of this lecture course, it was noted that in theanalysis of speech signals, the speech waveform is broken up intoshort segments whose duration is typically 10 to 20 milliseconds.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 15/55THEUN IV ERSITYOFEDI N BURGHQuasi-stationarityAt the introduction of this lecture course, it was noted that in theanalysis of speech signals, the speech waveform is broken up intoshort segments whose duration is typically 10 to 20 milliseconds.This is because speech can be modelled as a locally stationaryor quasi-stationary process.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 15/55THEUN IV ERSITYOFEDI N BURGHQuasi-stationarityAt the introduction of this lecture course, it was noted that in theanalysis of speech signals, the speech waveform is broken up intoshort segments whose duration is typically 10 to 20 milliseconds.This is because speech can be modelled as a locally stationaryor quasi-stationary process.Such processes possess statistical properties that change slowlyover short periods of time.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 15/55THEUN IV ERSITYOFEDI N BURGHQuasi-stationarityAt the introduction of this lecture course, it was noted that in theanalysis of speech signals, the speech waveform is broken up intoshort segments whose duration is typically 10 to 20 milliseconds.This is because speech can be modelled as a locally stationaryor quasi-stationary process.Such processes possess statistical properties that change slowlyover short periods of time.They are globally nonstationary, but are approximately locallystationary, and are modelled as if the statistics actually arestationary over a short segment of time.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 16/55THEUN IV ERSITYOFEDI N BURGHWSS PropertiesThe average power of a WSS process x(n) satisfies:rxx(0) = 2x + |x|2rxx(0) rxx(l), for all lStochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 16/55THEUN IV ERSITYOFEDI N BURGHWSS PropertiesThe average power of a WSS process x(n) satisfies:rxx(0) = 2x + |x|2rxx(0) rxx(l), for all lThe expression for power can be broken down as follows:Average DC Power: |x|2Average AC Power: 2xTotal average power: rxx(0)Total average power = Average DC power+Average AC powerStochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 16/55THEUN IV ERSITYOFEDI N BURGHWSS PropertiesThe average power of a WSS process x(n) satisfies:rxx(0) = 2x + |x|2rxx(0) rxx(l), for all lThe expression for power can be broken down as follows:Average DC Power: |x|2Average AC Power: 2xTotal average power: rxx(0)Total average power = Average DC power+Average AC powerMoreover, it follows that xx(0) xx(l).Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 16/55THEUN IV ERSITYOFEDI N BURGHWSS PropertiesIt is left as an exercise to show that the autocorrelation sequencerxx(l) is:a conjugate symmetric function of the lag l:rxx(l) = rxx(l)a nonnegative-definite or positive semi-definite function,such that for any sequence (n):Mn=1Mm=1(n) rxx(nm)(m) 0Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 16/55THEUN IV ERSITYOFEDI N BURGHWSS PropertiesIt is left as an exercise to show that the autocorrelation sequencerxx(l) is:a conjugate symmetric function of the lag l:rxx(l) = rxx(l)a nonnegative-definite or positive semi-definite function,such that for any sequence (n):Mn=1Mm=1(n) rxx(nm)(m) 0Note that, more generally, even a correlation function for anonstationary random process is positive semi-definite:Mn=1Mm=1(n) rxx(n,m)(m) 0 for any sequence (n)Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 17/55THEUN IV ERSITYOFEDI N BURGHEstimating statistical propertiesA stochastic process consists of the ensemble, x(n, ), and aprobability law, fX ({x} | {n}). If this information is availablen, the statistical properties are easily determined.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 17/55THEUN IV ERSITYOFEDI N BURGHEstimating statistical propertiesA stochastic process consists of the ensemble, x(n, ), and aprobability law, fX ({x} | {n}). If this information is availablen, the statistical properties are easily determined.In practice, only a limited number of realisations of a processis available, and often only one: i.e. {x(n, k), k {1, . . . , K}}is known for some K, but fX (x | n) is unknown.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 17/55THEUN IV ERSITYOFEDI N BURGHEstimating statistical propertiesA stochastic process consists of the ensemble, x(n, ), and aprobability law, fX ({x} | {n}). If this information is availablen, the statistical properties are easily determined.In practice, only a limited number of realisations of a processis available, and often only one: i.e. {x(n, k), k {1, . . . , K}}is known for some K, but fX (x | n) is unknown.Is is possible to infer the statistical characteristics of a processfrom a single realisation? Yes, for the following class ofsignals:Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 17/55THEUN IV ERSITYOFEDI N BURGHEstimating statistical propertiesA stochastic process consists of the ensemble, x(n, ), and aprobability law, fX ({x} | {n}). If this information is availablen, the statistical properties are easily determined.In practice, only a limited number of realisations of a processis available, and often only one: i.e. {x(n, k), k {1, . . . , K}}is known for some K, but fX (x | n) is unknown.Is is possible to infer the statistical characteristics of a processfrom a single realisation? Yes, for the following class ofsignals:ergodic processes;Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 17/55THEUN IV ERSITYOFEDI N BURGHEstimating statistical propertiesA stochastic process consists of the ensemble, x(n, ), and aprobability law, fX ({x} | {n}). If this information is availablen, the statistical properties are easily determined.In practice, only a limited number of realisations of a processis available, and often only one: i.e. {x(n, k), k {1, . . . , K}}is known for some K, but fX (x | n) is unknown.Is is possible to infer the statistical characteristics of a processfrom a single realisation? Yes, for the following class ofsignals:ergodic processes;nonstationary processes where additional structure aboutthe autocorrelation function is known (beyond the scope ofthis course).Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 18/55THEUN IV ERSITYOFEDI N BURGHEnsemble and Time-AveragesEnsemble averaging, as considered so far in the course, is notfrequently used in practice since it is impractical to obtain thenumber of realisations needed for an accurate estimate.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 18/55THEUN IV ERSITYOFEDI N BURGHEnsemble and Time-AveragesEnsemble averaging, as considered so far in the course, is notfrequently used in practice since it is impractical to obtain thenumber of realisations needed for an accurate estimate.A statistical average that can be obtained from a singlerealisation of a process is a time-average, defined by:g(x(n)) , limN12N + 1Nn=Ng(x(n))For every ensemble average, a corresponding time-average canbe defined; the above corresponds to: E [g(x(n))].Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 18/55THEUN IV ERSITYOFEDI N BURGHEnsemble and Time-AveragesEnsemble averaging, as considered so far in the course, is notfrequently used in practice since it is impractical to obtain thenumber of realisations needed for an accurate estimate.A statistical average that can be obtained from a singlerealisation of a process is a time-average, defined by:g(x(n)) , limN12N + 1Nn=Ng(x(n))For every ensemble average, a corresponding time-average canbe defined; the above corresponds to: E [g(x(n))].Time-averages are random variables since they implicitly dependon the particular realisation, given by . Averages ofdeterministic signals are fixed numbers or sequences, eventhough they are given by the same expression.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 19/55THEUN IV ERSITYOFEDI N BURGHErgodicityA stochastic process, x(n), is ergodic if its ensembleaverages can be estimated from a single realisation of aprocess using time averages.The two most important degrees of ergodicity are:Mean-Ergodic (or ergodic in the mean) processes have identicalexpected values and sample-means:x(n) = E [x(n)]Covariance-Ergodic Processes (or ergodic in correlation) have theproperty that:x(n)x(n l) = E [x(n)x(n l)]Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 19/55THEUN IV ERSITYOFEDI N BURGHErgodicityIt should be intuitiveness obvious that ergodic processes mustbe stationary and, moreover, that a process which is ergodicboth in the mean and correlation is WSS.WSS processes are not necessarily ergodic.Ergodic is often used to mean both ergodic in the mean andcorrelation.In practice, only finite records of data are available, andtherefore an estimate of the time-average will be given byg(x(n)) =1NnNg(x(n))where N is the number of data-points available. Theperformance of this estimator will be discussed later in thiscourse.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 20/55THEUN IV ERSITYOFEDI N BURGHJoint Signal StatisticsCross-correlation and cross-covariance A measure of thedependence between values of two different stochasticprocesses is given by the cross-correlation andcross-covariance functions:rxy(n1, n2) = E [x(n1) y(n2)]xy(n1, n2) = rxy(n1, n2) x(n1)y(n2)Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 20/55THEUN IV ERSITYOFEDI N BURGHJoint Signal StatisticsCross-correlation and cross-covariance A measure of thedependence between values of two different stochasticprocesses is given by the cross-correlation andcross-covariance functions:rxy(n1, n2) = E [x(n1) y(n2)]xy(n1, n2) = rxy(n1, n2) x(n1)y(n2)Normalised cross-correlation (or cross-covariance) Thecross-covariance provides a measure of similarity of thedeviation from the respective means of two processes. Itmakes sense to consider this deviation relative to theirstandard deviations; thus, normalised cross-correlations:xy(n1, n2) =xy(n1, n2)x(n1)y(n2)Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 21/55THEUN IV ERSITYOFEDI N BURGHTypes of Joint Stochastic ProcessesStatistically independence of two stochastic processes occurs when,for every nx and ny,fXY (x, y | nx, ny) = fX (x | nx) fY (y | ny)Uncorrelated stochastic processes have, for all nx & ny 6= nx:xy(nx, ny) = 0rxy(nx, ny) = x(nx)y(ny)Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 21/55THEUN IV ERSITYOFEDI N BURGHTypes of Joint Stochastic ProcessesStatistically independence of two stochastic processes occurs when,for every nx and ny,fXY (x, y | nx, ny) = fX (x | nx) fY (y | ny)Uncorrelated stochastic processes have, for all nx & ny 6= nx:xy(nx, ny) = 0rxy(nx, ny) = x(nx)y(ny)Joint stochastic processes that are statistically independent areuncorrelated, but not necessarily vice-versa, except for Gaussianprocesses. Nevertheless, a measure of uncorrelatedness is oftenused as a measure of independence. More on this later.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 21/55THEUN IV ERSITYOFEDI N BURGHTypes of Joint Stochastic ProcessesOrthogonal joint processes have, for every n1 and n2 6= n1:rxy(n1, n2) = 0Joint WSS is a similar to WSS for a single stochastic process, andis useful since it facilitates a spectral description, as discussedlater in this course:rxy(l) = rxy(n1 n2) = ryx(l) = E [x(n) y(n l)]xy(l) = xy(n1 n2) = yx(l) = rxy(l) x yJoint-Ergodicity applies to two ergodic processes, x(n) and y(n),whose ensemble cross-correlation can be estimated from atime-average:x(n) y(n l) = E [x(n) y(n l)]Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 22/55THEUN IV ERSITYOFEDI N BURGHCorrelation MatricesLet an M -dimensional random vector X(n, ) X(n) be derivedfrom the random process x(n) as follows:X(n) ,[x(n) x(n 1) x(nM + 1)]TStochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 22/55THEUN IV ERSITYOFEDI N BURGHCorrelation MatricesLet an M -dimensional random vector X(n, ) X(n) be derivedfrom the random process x(n) as follows:X(n) ,[x(n) x(n 1) x(nM + 1)]TThen its mean is given by an M -vectorX(n) ,[x(n) x(n 1) x(nM + 1)]TStochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 22/55THEUN IV ERSITYOFEDI N BURGHCorrelation MatricesLet an M -dimensional random vector X(n, ) X(n) be derivedfrom the random process x(n) as follows:X(n) ,[x(n) x(n 1) x(nM + 1)]TThen its mean is given by an M -vectorX(n) ,[x(n) x(n 1) x(nM + 1)]Tand the M M correlation matrix is given by:RX(n) ,rxx(n, n) rxx(n, nM + 1).... . ....rxx(nM + 1, n) rxx(nM + 1, nM + 1)Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 22/55THEUN IV ERSITYOFEDI N BURGHCorrelation MatricesFor stationary processes, the correlation matrix has an interestingadditional structure. Note that:1. RX(n) is a constant matrix RX;2. rxx(n i, n j) = rxx(j i) = rxx(l), l = j i;3. conjugate symmetry gives rxx(l) = rxx(l).Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 22/55THEUN IV ERSITYOFEDI N BURGHCorrelation MatricesFor stationary processes, the correlation matrix has an interestingadditional structure. Note that:1. RX(n) is a constant matrix RX;2. rxx(n i, n j) = rxx(j i) = rxx(l), l = j i;3. conjugate symmetry gives rxx(l) = rxx(l).Hence, the matrix Rxx is given by:RX ,rxx(0) rxx(1) rxx(2) rxx(M 1)rxx(1) rxx(0) rxx(1) rxx(M 2)rxx(2) rxx(1) rxx(0) rxx(M 3).......... . ....rxx(M 1) rxx(M 2) rxx(M 3) rxx(0)Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 23/55THEUN IV ERSITYOFEDI N BURGHMarkov ProcessesA powerful model for a stochastic process known as a Markovmodel is introduced; such a process that satisfies this model isknown as a Markov process.Quite simply, a Markov process is one in which the probability ofany particular value in a sequence is dependent upon thepreceding sample values.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 23/55THEUN IV ERSITYOFEDI N BURGHMarkov ProcessesA powerful model for a stochastic process known as a Markovmodel is introduced; such a process that satisfies this model isknown as a Markov process.Quite simply, a Markov process is one in which the probability ofany particular value in a sequence is dependent upon thepreceding sample values.The simplest kind of dependence arises when the probability ofany sample depends only upon the value of the immediatelypreceding sample, and this is known as a first-order Markovprocess.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 23/55THEUN IV ERSITYOFEDI N BURGHMarkov ProcessesA powerful model for a stochastic process known as a Markovmodel is introduced; such a process that satisfies this model isknown as a Markov process.Quite simply, a Markov process is one in which the probability ofany particular value in a sequence is dependent upon thepreceding sample values.The simplest kind of dependence arises when the probability ofany sample depends only upon the value of the immediatelypreceding sample, and this is known as a first-order Markovprocess.This simple process is a surprisingly good model for a number ofpractical signal processing, communications and controlproblems.Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 23/55THEUN IV ERSITYOFEDI N BURGHMarkov ProcessesAs an example of a Markov process, consider the processgenerated by the difference equationx(n) = a x(n 1) + w(n)where a is a known constant, and w(n) is a sequence ofzero-mean i. i. d. Gaussian random variables with variance 2Wdensity:fW (w(n)) =122Wexp{w(n)222W}Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 23/55THEUN IV ERSITYOFEDI N BURGHMarkov ProcessesAs an example of a Markov process, consider the processgenerated by the difference equationx(n) = a x(n 1) + w(n)where a is a known constant, and w(n) is a sequence ofzero-mean i. i. d. Gaussian random variables with variance 2Wdensity:fW (w(n)) =122Wexp{w(n)222W}The conditional density of x(n) given x(n 1) is also Gaussian,fX (x(n) | x(n 1)) =122Wexp{(x(n) + ax(n 1))222W}Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 23/55THEUN IV ERSITYOFEDI N BURGHMarkov ProcessesDefinition (Markov Process). A random process is a P th-orderMarkov process if the distribution of x(n), given the infinite past,depends only on the previous P samples{x(n 1), . . . , x(n P )}; that is, if:fX (x(n) | x(n 1), x(n 2), . . . ) = fX (x(n) | x(n 1), . . . , x(n P ))Stochastic ProcessesDefinition of a StochasticProcess Interpretation of SequencesPredictable ProcessesDescription using pdfsSecond-order StatisticalDescriptionExample of calculatingautocorrelationsTypes of StochasticProcessesStationary ProcessesOrder-N and strict-sensestationarityWide-sense stationarityWide-sensecyclo-stationarityQuasi-stationarityWSS PropertiesEstimating statisticalpropertiesEnsemble andTime-AveragesErgodicity Joint Signal StatisticsTypes of Joint StochasticProcessesCorrelation MatricesMarkov ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal Models- p. 23/55THEUN IV ERSITYOFEDI N BURGHMarkov ProcessesDefinition (Markov Process). A random process is a P th-orderMarkov process if the distribution of x(n), given the infinite past,depends only on the previous P samples{x(n 1), . . . , x(n P )}; that is, if:fX (x(n) | x(n 1), x(n 2), . . . ) = fX (x(n) | x(n 1), . . . , x(n P ))Finally, it is noted that if x(n) takes on a countable (discrete) setof values, a Markov random process is called a Markov chain.- p. 24/55Handout 3Power Spectral DensityStochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 25/55THEUN IV ERSITYOFEDI N BURGHIntroductionFrequency- and transform-domain methods are very powerfultools for the analysis of deterministic sequences. It seems naturalto extend these techniques to analysis stationary randomprocesses.Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 25/55THEUN IV ERSITYOFEDI N BURGHIntroductionFrequency- and transform-domain methods are very powerfultools for the analysis of deterministic sequences. It seems naturalto extend these techniques to analysis stationary randomprocesses.So far in this course, stationary stochastic processes have beenconsidered in the time-domain through the use of theautocorrelation function (ACF).Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 25/55THEUN IV ERSITYOFEDI N BURGHIntroductionFrequency- and transform-domain methods are very powerfultools for the analysis of deterministic sequences. It seems naturalto extend these techniques to analysis stationary randomprocesses.So far in this course, stationary stochastic processes have beenconsidered in the time-domain through the use of the ACF.Since the ACF for a stationary process is a function of asingle-discrete time process, then the question arises as to whatthe discrete-time Fourier transform (DTFT) corresponds to.Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 25/55THEUN IV ERSITYOFEDI N BURGHIntroductionFrequency- and transform-domain methods are very powerfultools for the analysis of deterministic sequences. It seems naturalto extend these techniques to analysis stationary randomprocesses.So far in this course, stationary stochastic processes have beenconsidered in the time-domain through the use of the ACF.Since the ACF for a stationary process is a function of asingle-discrete time process, then the question arises as to whatthe DTFT corresponds to.It turns out to be known as the power spectral density (PSD) ofa stationary random process, and the PSD is an extremelypowerful and conceptually appealing tool in statistical signalprocessing.Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 25/55THEUN IV ERSITYOFEDI N BURGHIntroductionIn signal theory for deterministic signals, spectra are used torepresent a function as a superposition of exponential functions.Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 25/55THEUN IV ERSITYOFEDI N BURGHIntroductionIn signal theory for deterministic signals, spectra are used torepresent a function as a superposition of exponential functions.For random signals, the notion of a spectrum has twointerpretations:Transform of averages The first involves transform of averages (ormoments). As will be seen, this will be the Fourier transformof the autocorrelation function.Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 25/55THEUN IV ERSITYOFEDI N BURGHIntroductionIn signal theory for deterministic signals, spectra are used torepresent a function as a superposition of exponential functions.For random signals, the notion of a spectrum has twointerpretations:Transform of averages The first involves transform of averages (ormoments). As will be seen, this will be the Fourier transformof the autocorrelation function.Stochastic decomposition The second interpretation represents astochastic process as a superposition of exponentials, wherethe coefficients are themselves random variables. Hence, x(n)can be represented as:x(n) =12X(ej) ejn d, n Rwhere X(ej) is a random variable for a given value of .Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 26/55THEUN IV ERSITYOFEDI N BURGHThe power spectral densityThe discrete-time Fourier transform of the autocorrelationfunction of a stationary stochastic process x[n, ] is known as thepower spectral density (PSD), is denoted by Pxx(ej), and isgiven by:Pxx(ej) =Zrxx[] ejwhere is frequency in radians per sample.Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 26/55THEUN IV ERSITYOFEDI N BURGHThe power spectral densityThe discrete-time Fourier transform of the autocorrelationfunction of a stationary stochastic process x[n, ] is known as thepower spectral density (PSD), is denoted by Pxx(ej), and isgiven by:Pxx(ej) =Zrxx[] ejwhere is frequency in radians per sample.The autocorrelation function, rxx[], can be recovered from thePSD by using the inverse-DTFT:rxx[] =12Pxx(ej) ej d, ZStochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 27/55THEUN IV ERSITYOFEDI N BURGHProperties of the power spectral densityPxx(ej) : R+; in otherwords, the PSD is real valued, andnonnegative definite. i.e.Pxx(ej) 0Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 27/55THEUN IV ERSITYOFEDI N BURGHProperties of the power spectral densityPxx(ej) : R+; in otherwords, the PSD is real valued, andnonnegative definite. i.e.Pxx(ej) 0Pxx(ej) = Pxx(ej(+2n)); in otherwords, the PSD is periodicwith period 2.Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 27/55THEUN IV ERSITYOFEDI N BURGHProperties of the power spectral densityPxx(ej) : R+; in otherwords, the PSD is real valued, andnonnegative definite. i.e.Pxx(ej) 0Pxx(ej) = Pxx(ej(+2n)); in otherwords, the PSD is periodicwith period 2.If x[n] is real-valued, then:Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 27/55THEUN IV ERSITYOFEDI N BURGHProperties of the power spectral densityPxx(ej) : R+; in otherwords, the PSD is real valued, andnonnegative definite. i.e.Pxx(ej) 0Pxx(ej) = Pxx(ej(+2n)); in otherwords, the PSD is periodicwith period 2.If x[n] is real-valued, then:rxx[] is real and even;Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 27/55THEUN IV ERSITYOFEDI N BURGHProperties of the power spectral densityPxx(ej) : R+; in otherwords, the PSD is real valued, andnonnegative definite. i.e.Pxx(ej) 0Pxx(ej) = Pxx(ej(+2n)); in otherwords, the PSD is periodicwith period 2.If x[n] is real-valued, then:rxx[] is real and even;Pxx(ej) = Pxx(ej) is an even function of .Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 27/55THEUN IV ERSITYOFEDI N BURGHProperties of the power spectral densityPxx(ej) : R+; in otherwords, the PSD is real valued, andnonnegative definite. i.e.Pxx(ej) 0Pxx(ej) = Pxx(ej(+2n)); in otherwords, the PSD is periodicwith period 2.If x[n] is real-valued, then:rxx[] is real and even;Pxx(ej) = Pxx(ej) is an even function of .The area under Pxx(ej) is nonnegative and is equal to theaverage power of x[n]. Hence:12Pxx(ej) d = rxx[0] = E[|x[n] |2] 0Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 28/55THEUN IV ERSITYOFEDI N BURGHGeneral form of the PSDA process, x(n), and therefore its corresponding autocorrelationfunction (ACF), rxx(l), can always be decomposed into azero-mean aperiodic component, r(a)xx (l), and a non-zero-meanperiodic component, r(p)xx (l):rxx(l) = r(a)xx (l) + r(p)xx (l)Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 28/55THEUN IV ERSITYOFEDI N BURGHGeneral form of the PSDA process, x(n), and therefore its corresponding autocorrelationfunction (ACF), rxx(l), can always be decomposed into azero-mean aperiodic component, r(a)xx (l), and a non-zero-meanperiodic component, r(p)xx (l):rxx(l) = r(a)xx (l) + r(p)xx (l)Theorem (PSD of a non-zero-mean process with periodic component).The most general definition of the PSD for a non-zero-meanstochastic process with a periodic component isPxx(ej) = P (a)xx (ej) +2KkKP (p)xx (k) ( k) P(a)xx (ej) is the DTFT of r(a)xx (l), while P(p)xx (k) are the discreteFourier transform (DFT) coefficients for r(p)xx (l) .Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 28/55THEUN IV ERSITYOFEDI N BURGHGeneral form of the PSDExample ( [Manolakis:2001, Harmonic Processes, Page 110-111]).Determine the PSD of the harmonic process defined by:x[n] =Mk=1Ak cos(kn+ k), k 6= 0Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 28/55THEUN IV ERSITYOFEDI N BURGHGeneral form of the PSDExample ( [Manolakis:2001, Harmonic Processes, Page 110-111]).Determine the PSD of the harmonic process defined by:x[n] =Mk=1Ak cos(kn+ k), k 6= 0SOLUTION. x[n] is a stationary process with zero-mean, andautocorrelation function (ACF):rxx[] =12Mk=1|Ak|2 cosk, < < Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 28/55THEUN IV ERSITYOFEDI N BURGHGeneral form of the PSDExample ( [Manolakis:2001, Harmonic Processes, Page 110-111]).Determine the PSD of the harmonic process defined by:x[n] =Mk=1Ak cos(kn+ k), k 6= 0SOLUTION. Hence, the ACF can be written as:rxx[] =Mk=M|Ak|24ejk, < < where the following are defined: A0 = 0, Ak = Ak, andk = k.Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 28/55THEUN IV ERSITYOFEDI N BURGHGeneral form of the PSDExample ( [Manolakis:2001, Harmonic Processes, Page 110-111]).Determine the PSD of the harmonic process defined by:x[n] =Mk=1Ak cos(kn+ k), k 6= 0SOLUTION. Hence, the ACF can be written as:rxx[] =Mk=M|Ak|24ejk, < < where the following are defined: A0 = 0, Ak = Ak, andk = k.Hence, it directly followsPxx(ej) = 2Mk=M|Ak|24(k) =2Mk=M|Ak|2(k) Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 29/55THEUN IV ERSITYOFEDI N BURGHThe cross-power spectral densityThe cross-power spectral density (CPSD) of two jointly stationarystochastic processes, x(n) and y(n), provides a description oftheir statistical relations in the frequency domain.It is defined, naturally, as the DTFT of the cross-correlation,rxy() , E [x(n) y(n )]:Pxy(ej) = F{rxy()} =lZrxy() ejStochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 29/55THEUN IV ERSITYOFEDI N BURGHThe cross-power spectral densityThe cross-power spectral density (CPSD) of two jointly stationarystochastic processes, x(n) and y(n), provides a description oftheir statistical relations in the frequency domain.It is defined, naturally, as the DTFT of the cross-correlation,rxy() , E [x(n) y(n )]:Pxy(ej) = F{rxy()} =lZrxy() ejThe cross-correlation rxy(l) can be recovered by using theinverse-DTFT:rxy(l) =12Pxy(ej) ejl d, l RStochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 29/55THEUN IV ERSITYOFEDI N BURGHThe cross-power spectral densityThe cross-power spectral density (CPSD) of two jointly stationarystochastic processes, x(n) and y(n), provides a description oftheir statistical relations in the frequency domain.It is defined, naturally, as the DTFT of the cross-correlation,rxy() , E [x(n) y(n )]:Pxy(ej) = F{rxy()} =lZrxy() ejThe cross-correlation rxy(l) can be recovered by using theinverse-DTFT:rxy(l) =12Pxy(ej) ejl d, l RThe cross-spectrum Pxy(ej) is, in general, a complex function of.Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 29/55THEUN IV ERSITYOFEDI N BURGHThe cross-power spectral densitySome properties of the CPSD and related definitions include:1. Pxy(ej) is periodic in with period 2.2. Since rxy(l) = ryx(l), then it follows:Pxy(ej) = P yx(ej)3. If the process x(n) is real, then rxy(l) is real, and:Pxy(ej) = P xy(ej)4. The coherence function, is given by:xy(ej) ,Pxy(ej)Pxx(ej)Pyy(ej)Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 30/55THEUN IV ERSITYOFEDI N BURGHComplex Spectral Density FunctionsThe second moment quantities that described a random processin the transform domain are known as the complex spectraldensity and complex cross-spectral density functions.Stochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 30/55THEUN IV ERSITYOFEDI N BURGHComplex Spectral Density FunctionsThe second moment quantities that described a random processin the transform domain are known as the complex spectraldensity and complex cross-spectral density functions.Hence, rxx(l)z Pxx(z) and rxy(l)z Pxy(z), where:Pxx(z) =lZrxx(l) zlPxy(z) =lZrxy(l) zlStochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 30/55THEUN IV ERSITYOFEDI N BURGHComplex Spectral Density FunctionsThe second moment quantities that described a random processin the transform domain are known as the complex spectraldensity and complex cross-spectral density functions.Hence, rxx(l)z Pxx(z) and rxy(l)z Pxy(z), where:Pxx(z) =lZrxx(l) zlPxy(z) =lZrxy(l) zlIf the unit circle, defined by z = ej is within the region ofconvergence of these summations, then:Pxx(ej) = Pxx(z)|z=ejPxy(ej) = Pxy(z)|z=ejStochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 30/55THEUN IV ERSITYOFEDI N BURGHComplex Spectral Density FunctionsThe inverse of the complex spectral and cross-spectral densitiesare given by the contour integral:rxx(l) =12jCPxx(z) zl1 dzrxy(l) =12jCPxy(z) zl1 dzStochastic ProcessesPower Spectral Density IntroductionThe power spectraldensityProperties of the powerspectral densityGeneral form of the PSDThe cross-power spectraldensityComplex Spectral DensityFunctionsLinear Systems TheoryLinear Signal Models- p. 30/55THEUN IV ERSITYOFEDI N BURGHComplex Spectral Density FunctionsThe inverse of the complex spectral and cross-spectral densitiesare given by the contour integral:rxx(l) =12jCPxx(z) zl1 dzrxy(l) =12jCPxy(z) zl1 dzSome properties of the complex spectral densities include:1. Conjugate-symmetry:Pxx(z) = Pxx(1/z) and Pxy(z) = Pxy(1/z)2. For the case when x(n) is real, then:Pxx(z) = Pxx(z1)- p. 31/55Handout 4Linear Systems TheoryStochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of alinear time-invariant (LTI)SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 32/55THEUN IV ERSITYOFEDI N BURGHSystems with Stochastic InputsA graphical representation of a random process at theoutput of a system in relation to a random process at theinput of the system.What does it mean to apply a stochastic signal to the input of asystem?Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 32/55THEUN IV ERSITYOFEDI N BURGHSystems with Stochastic InputsIn principle, the statistics of the output of a system can beexpressed in terms of the statistics of the input. However, ingeneral this is a complicated problem except in special cases.Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 32/55THEUN IV ERSITYOFEDI N BURGHSystems with Stochastic InputsIn principle, the statistics of the output of a system can beexpressed in terms of the statistics of the input. However, ingeneral this is a complicated problem except in special cases.A special case is that of linear systems, and this is considerednext.Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 33/55THEUN IV ERSITYOFEDI N BURGHLTI Systems with Stationary InputsSince each sequence (realisation) of a stochastic process is adeterministic signal, there is a well-defined input signalproducing a well-defined output signal corresponding to a singlerealisation of the output stochastic process:y(n, ) =k=h(k)x(n k, )Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 33/55THEUN IV ERSITYOFEDI N BURGHLTI Systems with Stationary InputsSince each sequence (realisation) of a stochastic process is adeterministic signal, there is a well-defined input signalproducing a well-defined output signal corresponding to a singlerealisation of the output stochastic process:y(n, ) =k=h(k)x(n k, )A complete description of y(n, ) requires the computation ofan infinite number of convolutions, corresponding to eachvalue of .Thus, a better description would be to consider the statisticalproperties of y(n, ) in terms of the statistical properties of theinput and the characteristics of the system.Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 33/55THEUN IV ERSITYOFEDI N BURGHLTI Systems with Stationary InputsTo investigate the statistical input-output properties of a linearsystem, note the following fundamental theorem:Theorem (Expectation in Linear Systems). For any linear system,E [L[x(n)]] = L[E [x(n)]]In other words, the mean y(n) of the output y(n) equals theresponse of the system to the mean x(n) of the input:y(n) = L[x(n)] Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 34/55THEUN IV ERSITYOFEDI N BURGHInput-output Statistics of a LTI SystemIf a stationary stochastic process x[n] with mean value x andcorrelation rxx[] is applied to the input of a LTI system withimpulse response h[n] and transfer function H(ej), then the:Output mean value is given by:y = xk=h[k] = x H(ej0)Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 34/55THEUN IV ERSITYOFEDI N BURGHInput-output Statistics of a LTI SystemIf a stationary stochastic process x[n] with mean value x andcorrelation rxx[] is applied to the input of a LTI system withimpulse response h[n] and transfer function H(ej), then the:Output mean value is given by:y = xk=h[k] = x H(ej0)Input-output cross-correlation is given by:rxy[] = h[] rxx[] =k=h[k] rxx[ k]Similarly, it follows that ryx(l) = h(l) rxx(l).Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 34/55THEUN IV ERSITYOFEDI N BURGHInput-output Statistics of a LTI SystemOutput autocorrelation is obtained by pre-multiplying thesystem-output by y(n l) and taking expectations:ryy(l) =k=h(k)E [x(n k) y(n l)] = h(l) rxy(l)Substituting the expression for rxy(l) gives:ryy(l) = h(l) h(l) rxx(l) = rhh(l) rxx(l)An equivalent LTI system for autocorrelation filtration.Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 34/55THEUN IV ERSITYOFEDI N BURGHInput-output Statistics of a LTI SystemOutput-power of the process y(n) is given by ryy(0) = E[|y(n)|2],and therefore since ryy(l) = rhh(l) rxx(l),Noting power, Pyy, is real, then taking complex-conjugates usingrxx(l) = rxx(l):Pyy =k= rhh(k) rxx(k) =n= h(n)k= rxx(n+ k)h(k)Output pdf In general, it is very difficult to calculate the pdf of theoutput of a LTI system, except in special cases, namelyGaussian processes.Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 35/55THEUN IV ERSITYOFEDI N BURGHSystem identificationSystem identification by cross-correlation.The system is excited with a white Gaussian noise (WGN) inputwith autocorrelation function:rxx(l) = (l)Since the output-input cross-correlation can be written as:ryx(l) = h(l) rxx(l)then, with rxx(l) = (l), it follows:ryx(l) = h(l) (l) = h(l)Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 36/55THEUN IV ERSITYOFEDI N BURGHLTV Systems with Nonstationary InputsGeneral LTV system with nonstationary inputThe input and output are related by the generalised convolution:y(n) =k=h(n, k)x(k)where h(n, k) is the response at time-index n to an impulseoccurring at the system input at time-index k.Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 36/55THEUN IV ERSITYOFEDI N BURGHLTV Systems with Nonstationary InputsGeneral LTV system with nonstationary inputThe input and output are related by the generalised convolution:y(n) =k=h(n, k)x(k)where h(n, k) is the response at time-index n to an impulseoccurring at the system input at time-index k.The mean, autocorrelation and autocovariance sequences of theoutput, y(n), as well as the cross-correlation and cross-covariancefunctions between the input and the output, can be calculated ina similar way as for LTI systems with stationary inputs.Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 37/55THEUN IV ERSITYOFEDI N BURGHDifference EquationConsider a LTI system that can be represented by a differenceequation:Pp=0ap y(n p) =Qq=0bq x(n q)where a0 , 1.Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 37/55THEUN IV ERSITYOFEDI N BURGHDifference EquationConsider a LTI system that can be represented by a differenceequation:Pp=0ap y(n p) =Qq=0bq x(n q)where a0 , 1.Assuming that both x(n) and y(n) are stationary processes, thentaking expectations of both sides gives,y =Qq=0 bq1 +Pp=1 apxStochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 37/55THEUN IV ERSITYOFEDI N BURGHDifference EquationNext, multiplying the system equation throughout by y(m) andtaking expectations gives:Pp=0ap ryy(n p,m) =Qq=0bq rxy(n q,m)Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 37/55THEUN IV ERSITYOFEDI N BURGHDifference EquationNext, multiplying the system equation throughout by y(m) andtaking expectations gives:Pp=0ap ryy(n p,m) =Qq=0bq rxy(n q,m)Similarly, instead multiply though by x(m) to give:Pp=0ap ryx(n p,m) =Qq=0bq rxx(n q,m)These two difference equations may be used to solve forryy(n1, n2) and rxy(n1, n2). Similar expressions can be obtainedfor the covariance functions.Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 37/55THEUN IV ERSITYOFEDI N BURGHDifference EquationExample ( [Manolakis:2000, Example 3.6.2, Page 141]). Let x(n) be arandom process generated by the first order difference equationgiven by:x(n) = x(n 1) + w(n), || 1, n Z where w(n) N(w, 2w)is an i. i. d. WGN process.Demonstrate that the process x(n) is stationary, anddetermine the mean x.Determine the autocovariance and autocorrelation function,xx(l) and rxx(l).Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 38/55THEUN IV ERSITYOFEDI N BURGHFrequency-Domain Analysis of LTI systemsThe PSD at the input and output of a LTI system withstationary input.Pxy(ej) = H(ej)Pxx(ej)Pyx(ej) = H(ej)Pxx(ej)Pyy(ej) = |H(ej)|2 Pxx(ej)Stochastic ProcessesPower Spectral DensityLinear Systems TheorySystems with StochasticInputsLTI Systems withStationary Inputs Input-output Statistics of aLTI SystemSystem identificationLTV Systems withNonstationary InputsDifference EquationFrequency-DomainAnalysis of LTI systemsLinear Signal Models- p. 38/55THEUN IV ERSITYOFEDI N BURGHFrequency-Domain Analysis of LTI systemsThe PSD at the input and output of a LTI system withstationary input.Pxy(ej) = H(ej)Pxx(ej)Pyx(ej) = H(ej)Pxx(ej)Pyy(ej) = |H(ej)|2 Pxx(ej)If the input and output autocorrelations or autospectraldensities are known, the magnitude response of a system|H(ej)| can be determined, but not the phase response.- p. 39/55Handout 5Linear Signal ModelsStochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 40/55THEUN IV ERSITYOFEDI N BURGHAbstractIn the last lecture, the response of a linear-system when astochastic process is applied at the input was considered.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 40/55THEUN IV ERSITYOFEDI N BURGHAbstractIn the last lecture, the response of a linear-system when astochastic process is applied at the input was considered.This lecture looks at the special class of stationary signals thatare obtained by driving a LTI system with white noise. Aparticular focus is placed on system functions that arerational; that is, they can be expressed at the ratio of twopolynomials.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 40/55THEUN IV ERSITYOFEDI N BURGHAbstractIn the last lecture, the response of a linear-system when astochastic process is applied at the input was considered.This lecture looks at the special class of stationary signals thatare obtained by driving a LTI system with white noise. Aparticular focus is placed on system functions that arerational; that is, they can be expressed at the ratio of twopolynomials.The following models are considered in detail:Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 40/55THEUN IV ERSITYOFEDI N BURGHAbstractIn the last lecture, the response of a linear-system when astochastic process is applied at the input was considered.This lecture looks at the special class of stationary signals thatare obtained by driving a LTI system with white noise. Aparticular focus is placed on system functions that arerational; that is, they can be expressed at the ratio of twopolynomials.The following models are considered in detail:All-pole systems and autoregressive (AR) processes;Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 40/55THEUN IV ERSITYOFEDI N BURGHAbstractIn the last lecture, the response of a linear-system when astochastic process is applied at the input was considered.This lecture looks at the special class of stationary signals thatare obtained by driving a LTI system with white noise. Aparticular focus is placed on system functions that arerational; that is, they can be expressed at the ratio of twopolynomials.The following models are considered in detail:All-pole systems and autoregressive (AR) processes;All-zero systems and moving average (MA) processes;Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 40/55THEUN IV ERSITYOFEDI N BURGHAbstractIn the last lecture, the response of a linear-system when astochastic process is applied at the input was considered.This lecture looks at the special class of stationary signals thatare obtained by driving a LTI system with white noise. Aparticular focus is placed on system functions that arerational; that is, they can be expressed at the ratio of twopolynomials.The following models are considered in detail:All-pole systems and autoregressive (AR) processes;All-zero systems and moving average (MA) processes;and pole-zero systems and autoregressive movingaverage (ARMA) processes.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 40/55THEUN IV ERSITYOFEDI N BURGHAbstractIn the last lecture, the response of a linear-system when astochastic process is applied at the input was considered.This lecture looks at the special class of stationary signals thatare obtained by driving a LTI system with white noise. Aparticular focus is placed on system functions that arerational; that is, they can be expressed at the ratio of twopolynomials.The following models are considered in detail:All-pole systems and autoregressive (AR) processes;All-zero systems and moving average (MA) processes;and pole-zero systems and autoregressive movingaverage (ARMA) processes.Pole-zero models are widely used for modelling stationarysignals with short memory; the concepts will be extended, inoverview at least, to nonstationary processes.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 41/55THEUN IV ERSITYOFEDI N BURGHThe Ubiquitous WGN SequenceThe simplest random signal model is the WSS WGN sequence:w(n) N(0, 2w)The sequence is i. i. d., and Pww(ej) = 2w, < .Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 42/55THEUN IV ERSITYOFEDI N BURGHFiltration of WGNBy filtering a WGN through a stable LTI system, it is possible toobtain a stochastic signal at the output with almost any arbitraryaperiodic correlation function or continuous PSD.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 42/55THEUN IV ERSITYOFEDI N BURGHFiltration of WGNBy filtering a WGN through a stable LTI system, it is possible toobtain a stochastic signal at the output with almost any arbitraryaperiodic correlation function or continuous PSD.Random signals with line PSDs can be generated by using theharmonic process model, which is a linear combination ofsinusoidal sequences with statistically independent randomphases. Signal models with mixed PSDs can be obtained bycombining these two models; a process justified by the Wolddecomposition.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 42/55THEUN IV ERSITYOFEDI N BURGHFiltration of WGNPww(e )jws2w-p +p wwP kww( )-p +pWhite harmonicprocessWhitenoiseInputexcitationw( ) ( , )n w n= z00H(e )jw-p +p w0LTI SystemH z D z A z A z( ) or ( )/ ( ) or 1/ ( )Desiredsignalx( ) ( , )n x n= z-p +p w0Pxx(e )jw-p +p w0P kxx( )Signal models with continuous and discrete (line) powerspectrum densities.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 42/55THEUN IV ERSITYOFEDI N BURGHFiltration of WGNThe speech synthesis model.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 43/55THEUN IV ERSITYOFEDI N BURGHNonparametric and parametric modelsNonparametric models have no restriction on its form, or thenumber of parameters characterising the model. For example,specifying a LTI filter by its impulse response is anonparametric model.Parametric models, on the other hand, describe a system with afinite number of parameters. For example, if a LTI filter isspecified by a finite-order rational system function, it is aparametric model.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 43/55THEUN IV ERSITYOFEDI N BURGHNonparametric and parametric modelsNonparametric models have no restriction on its form, or thenumber of parameters characterising the model. For example,specifying a LTI filter by its impulse response is anonparametric model.Parametric models, on the other hand, describe a system with afinite number of parameters. For example, if a LTI filter isspecified by a finite-order rational system function, it is aparametric model.Two important analysis tools present themselves forparametric modelling:1. given the model parameters, analyse the characteristics ofthat model (in terms of moments etc.);2. design of a parametric system model to produce a randomsignal with a specified autocorrelation function or PSD.This is signal modelling.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 44/55THEUN IV ERSITYOFEDI N BURGHParametric Pole-Zero Signal ModelsConsider a system described by the following linearconstant-coefficient difference equation:x(n) = Pk=1ak x(n k) +Qk=0dk w(n k)Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 44/55THEUN IV ERSITYOFEDI N BURGHParametric Pole-Zero Signal ModelsConsider a system described by the following linearconstant-coefficient difference equation:x(n) = Pk=1ak x(n k) +Qk=0dk w(n k)Taking z-transforms gives the system function:H(z) =X(z)W (z)=Qk=0 dk zk1 +Pk=1 ak zk,D(z)A(z)= GQk=1(1 zk z1)Pk=1(1 pk z1)This system has Q zeros, {zk, k Q} where Q = {1, . . . , Q}, andP poles, {pk, k P}. The term G is the system gain.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 45/55THEUN IV ERSITYOFEDI N BURGHTypes of pole-zero modelsAll-pole model when Q = 0. The input-output difference equationis given by:x(n) = Pk=1ak x(n k) + d0 w(n)All-zero model when P = 0. The input-output relation is given by:x(n) =Qk=0dk w(n k)Pole-zero model when P > 0 and Q > 0.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 45/55THEUN IV ERSITYOFEDI N BURGHTypes of pole-zero modelsDifferent types of linear modelStochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 45/55THEUN IV ERSITYOFEDI N BURGHTypes of pole-zero modelsIf a parametric model is excited with WGN, the resulting outputsignal has second-order moments determined by the parametersof the model.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 45/55THEUN IV ERSITYOFEDI N BURGHTypes of pole-zero modelsIf a parametric model is excited with WGN, the resulting outputsignal has second-order moments determined by the parametersof the model.These stochastic processes have special names in the literature,and are known as:a moving average (MA) process when it is the output of an all-zeromodel;an autoregressive (AR) process when it is the output of an all-polemodel;an autoregressive moving average (ARMA) process when it is theoutput of an pole-zero model;each subject to a WGN process at the input.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 46/55THEUN IV ERSITYOFEDI N BURGHAll-pole ModelsAll-pole models are frequently used in signal processingapplications since they are:mathematically convenient since model parameters can beestimated by solving a set of linear equations, andthey widely parsimoniously approximate rational transferfunctions, especially resonant systems.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 46/55THEUN IV ERSITYOFEDI N BURGHAll-pole ModelsAll-pole models are frequently used in signal processingapplications since they are:mathematically convenient since model parameters can beestimated by solving a set of linear equations, andthey widely parsimoniously approximate rational transferfunctions, especially resonant systems.There are various model properties of the all-pole model that areuseful; these include:1. the systems impulse response;2. the autocorrelation of the impulse response;3. and minimum-phase conditions.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 47/55THEUN IV ERSITYOFEDI N BURGHFrequency Response of an All-Pole FilterThe all-pole model has form:H(z) =d0A(z)=d01 +Pk=1 ak zk=d0Pk=1(1 pk z1)and therefore its frequency response is given by:H(ej) =d01 +Pk=1 ak ejk=d0Pk=1(1 pk ej)Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 47/55THEUN IV ERSITYOFEDI N BURGHFrequency Response of an All-Pole FilterThe all-pole model has form:H(z) =d0A(z)=d01 +Pk=1 ak zk=d0Pk=1(1 pk z1)and therefore its frequency response is given by:H(ej) =d01 +Pk=1 ak ejk=d0Pk=1(1 pk ej)When each of the poles are written in the form pk = rkejk , thenthe frequency response can be written as:H(ej) =d0Pk=1(1 rk ej(k))Hence, it can be deduced that resonances occur near thefrequencies corresponding to the phase position of the poles.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 47/55THEUN IV ERSITYOFEDI N BURGHFrequency Response of an All-Pole FilterHence, the PSD of the output of an all-pole filter is given by:Pxx(ej) = 2wH(ej)2=G2Pk=11 rk ej(k)2where G = w d0 is the overall gain of the system.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 47/55THEUN IV ERSITYOFEDI N BURGHFrequency Response of an All-Pole FilterHence, the PSD of the output of an all-pole filter is given by:Pxx(ej) = 2wH(ej)2=G2Pk=11 rk ej(k)2where G = w d0 is the overall gain of the system.Consider the all-pole model with poles at positions:{pk} = {rk ejk} where{{rk} = {0.985, 0.951, 0.942, 0.933}{k} = 2 {270, 550, 844, 1131}/2450;Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 47/55THEUN IV ERSITYOFEDI N BURGHFrequency Response of an All-Pole Filter0 0.2 0.4 0.6 0.8 10123456 / |H(ej)|AllPole Magnitude Frequency Response 0.2 0.4 0.6 0.8 1302106024090270120300150330180 0AllPole Pole PositionsRe(z)Im(z)The frequency response and position of the poles in an all-polesystem.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 47/55THEUN IV ERSITYOFEDI N BURGHFrequency Response of an All-Pole Filter0 0.2 0.4 0.6 0.8 12015105051015 / 10 log 10 |Pxx(ej)|AllPole Power SpectrumPower spectral response of an all-pole model.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 48/55THEUN IV ERSITYOFEDI N BURGHImpulse Response of an All-Pole FilterThe impulse response of the all-pole filter satisfies the equation:h(n) = Pk=1ak h(n k) + d0 (n)If H(z) has its poles inside the unit circle, then h(n) is a causal,stable sequence, and the system is minimum-phase.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 48/55THEUN IV ERSITYOFEDI N BURGHImpulse Response of an All-Pole FilterThe impulse response of the all-pole filter satisfies the equation:h(n) = Pk=1ak h(n k) + d0 (n)If H(z) has its poles inside the unit circle, then h(n) is a causal,stable sequence, and the system is minimum-phase.Assuming causality, such that h(n) = 0, n < 0 then it followsh(k) = 0, k > 0, and therefore:h(n) =0 if n < 0d0 if n = 0Pk=1 ak h(n k) if n > 0Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 49/55THEUN IV ERSITYOFEDI N BURGHAll-Pole Modelling and Linear PredictionA linear predictor forms an estimate, or prediction, x(n), of thepresent value of a stochastic process x(n) from a linearcombination of the past P samples; that is:x(n) = Pk=1ak x(n k)Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 49/55THEUN IV ERSITYOFEDI N BURGHAll-Pole Modelling and Linear PredictionA linear predictor forms an estimate, or prediction, x(n), of thepresent value of a stochastic process x(n) from a linearcombination of the past P samples; that is:x(n) = Pk=1ak x(n k)The coefficients {ak} of the linear predictor are determined byattempting to minimise some function of the prediction errorgiven by:e(n) = x(n) x(n)Usually the objective function is equivalent to mean-squarederror (MSE), given by E =n e2(n).Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 49/55THEUN IV ERSITYOFEDI N BURGHAll-Pole Modelling and Linear PredictionHence, the prediction error can be written as:e(n) = x(n) x(n) = x(n) +Pk=1ak x(n k)Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 49/55THEUN IV ERSITYOFEDI N BURGHAll-Pole Modelling and Linear PredictionHence, the prediction error can be written as:e(n) = x(n) x(n) = x(n) +Pk=1ak x(n k)Thus, the prediction error is equal to the excitation of theall-pole model; e(n) = w(n). Clearly, finite impulseresponse (FIR) linear prediction and all-pole modelling areclosely related.Many of the properties and algorithms developed for eitherlinear prediction or all-pole modelling can be applied to theother.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 49/55THEUN IV ERSITYOFEDI N BURGHAll-Pole Modelling and Linear PredictionHence, the prediction error can be written as:e(n) = x(n) x(n) = x(n) +Pk=1ak x(n k)Thus, the prediction error is equal to the excitation of theall-pole model; e(n) = w(n). Clearly, FIR linear predictionand all-pole modelling are closely related.Many of the properties and algorithms developed for eitherlinear prediction or all-pole modelling can be applied to theother.To all intents and purposes, linear prediction, all-polemodelling, and AR processes (discussed next) are equivalentterms for the same concept.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 50/55THEUN IV ERSITYOFEDI N BURGHAutoregressive ProcessesWhile all-pole models refer to the properties of a rationalsystem containing only poles, AR processes refer to the resultingstochastic process that occurs as the result of WGN being appliedto the input of an all-pole filter.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 50/55THEUN IV ERSITYOFEDI N BURGHAutoregressive ProcessesWhile all-pole models refer to the properties of a rationalsystem containing only poles, AR processes refer to the resultingstochastic process that occurs as the result of WGN being appliedto the input of an all-pole filter.As such, the same input-output equations for all-pole models stillapply.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 50/55THEUN IV ERSITYOFEDI N BURGHAutoregressive ProcessesWhile all-pole models refer to the properties of a rationalsystem containing only poles, AR processes refer to the resultingstochastic process that occurs as the result of WGN being appliedto the input of an all-pole filter.As such, the same input-output equations for all-pole models stillapply.Thus:x(n) = Pk=1ak x(n k) + w(n), w(n) N(0, 2w)The autoregressive output, x(n), is a stationary sequence with amean value of zero, x = 0.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 50/55THEUN IV ERSITYOFEDI N BURGHAutoregressive ProcessesWhile all-pole models refer to the properties of a rationalsystem containing only poles, AR processes refer to the resultingstochastic process that occurs as the result of WGN being appliedto the input of an all-pole filter.Thus:x(n) = Pk=1ak x(n k) + w(n), w(n) N(0, 2w)The autoregressive output, x(n), is a stationary sequence with amean value of zero, x = 0.The autocorrelation function (ACF) can be calculated in a similarapproach to finding the output autocorrelation andcross-correlation for linear systems.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 50/55THEUN IV ERSITYOFEDI N BURGHAutoregressive ProcessesMultiply the difference through by x(n l) and takeexpectations to obtain:rxx(l) +Pk=1ak rxx(l k) = rwx(l)Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 50/55THEUN IV ERSITYOFEDI N BURGHAutoregressive ProcessesMultiply the difference through by x(n l) and takeexpectations to obtain:rxx(l) +Pk=1ak rxx(l k) = rwx(l)Observing that x(n) cannot depend on future values of w(n)since the system is causal, then rwx(l) = E [w(n)x(n l)] iszero if l > 0, and 2w if l = 0.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 50/55THEUN IV ERSITYOFEDI N BURGHAutoregressive ProcessesMultiply the difference through by x(n l) and takeexpectations to obtain:rxx(l) +Pk=1ak rxx(l k) = rwx(l)Thus, writing Equation ?? for l = {0, 1, . . . , P}matrix-vector form (noting that rxx(l) = rxx(l) and that theparameters {ak} are real) as:rxx(0) rxx(1) rxx(P )rxx(1) rxx(0) rxx(P 1)....... . ....rxx(P ) rxx(P 1) rxx(0)1a1...aP=2w0...0Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 51/55THEUN IV ERSITYOFEDI N BURGHAll-Zero modelsWhereas all-pole models can capture resonant features of aparticular PSD, it cannot capture nulls in the frequency response.These can only be modelled using a pole-zero or all-zero model.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 51/55THEUN IV ERSITYOFEDI N BURGHAll-Zero modelsWhereas all-pole models can capture resonant features of aparticular PSD, it cannot capture nulls in the frequency response.These can only be modelled using a pole-zero or all-zero model.The output of an all-zero model is the weighted average ofdelayed versions of the input signal. Thus, assume an all-zeromodel of the form:x(n) =Qk=0dk w(n k)where Q is the order of the model, and the corresponding systemfunction is given by:H(z) = D(z) =Qk=0dk zkStochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 52/55THEUN IV ERSITYOFEDI N BURGHFrequency Response of an All-Zero FilterThe all-zero model has form:H(z) = D(z) =Qk=0dk zk = d0Qk=1(1 zk z1)Therefore, its frequency response is given by:H(ej) =Qk=0dk ejk = d0Qk=1(1 zk ej)Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 52/55THEUN IV ERSITYOFEDI N BURGHFrequency Response of an All-Zero FilterThe all-zero model has form:H(z) = D(z) =Qk=0dk zk = d0Qk=1(1 zk z1)Therefore, its frequency response is given by:H(ej) =Qk=0dk ejk = d0Qk=1(1 zk ej)When each of the zeros are written in the form zk = rkejk , thenthe frequency response can be written as:H(ej) = d0Qk=1(1 rk ej(k))Hence, it can be deduced that troughs or nulls occur nearfrequencies corresponding to the phase position of the zeros.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 52/55THEUN IV ERSITYOFEDI N BURGHFrequency Response of an All-Zero FilterHence, the PSD of the output of an all-zero filter is given by:Pxx(ej) = 2wH(ej)2= G2Qk=11 rk ej(k)2where G = w d0 is the overall gain of the system.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 52/55THEUN IV ERSITYOFEDI N BURGHFrequency Response of an All-Zero FilterHence, the PSD of the output of an all-zero filter is given by:Pxx(ej) = 2wH(ej)2= G2Qk=11 rk ej(k)2where G = w d0 is the overall gain of the system.Consider the all-zero model with zeros at positions:{zk} = {rk ejk} where{{rk} = {0.985, 1, 0.942, 0.933}{k} = 2 {270, 550, 844, 1131}/2450;Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 52/55THEUN IV ERSITYOFEDI N BURGHFrequency Response of an All-Zero Filter0 0.2 0.4 0.6 0.8 10246810 / |H(ej)|AllZero Model Magnitude Frequency Response 0.2 0.4 0.6 0.8 1302106024090270120300150330180 0All-Zero Model Zero PositionsRe(z)Im(z)The frequency response and position of the zeros in an all-zerosystem.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 52/55THEUN IV ERSITYOFEDI N BURGHFrequency Response of an All-Zero Filter0 0.2 0.4 0.6 0.8 180604020020 / 10 log 10 |Pxx(ej)|AllZero Model Power SpectrumPower spectral response of an all-zero model.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 53/55THEUN IV ERSITYOFEDI N BURGHMoving-average processesA MA process refers to the stochastic process that is obtained atthe output of an all-zero filter when a WGN sequence is appliedto the input.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 53/55THEUN IV ERSITYOFEDI N BURGHMoving-average processesA MA process refers to the stochastic process that is obtained atthe output of an all-zero filter when a WGN sequence is appliedto the input.Thus, a MA process is an AZ(Q) model with d0 = 1 driven byWGN.x[n] = w[n] +Qk=1dk w[n k] , w[n] N(0, 2w)Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 53/55THEUN IV ERSITYOFEDI N BURGHMoving-average processesA MA process refers to the stochastic process that is obtained atthe output of an all-zero filter when a WGN sequence is appliedto the input.Thus, a MA process is an AZ(Q) model with d0 = 1 driven byWGN.x[n] = w[n] +Qk=1dk w[n k] , w[n] N(0, 2w)The output x(n) has zero-mean, and variance of2x = 2w[1 +Qk=1|dk|2]The autocorrelation function is given by:rxx[] = 2wrhh[] = 2wQdk+l dk, for 0 QStochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 54/55THEUN IV ERSITYOFEDI N BURGHPole-Zero ModelsThe output of a causal pole-zero model is given by the recursiveinput-output relationship:x[n] = Pk=1ak x[n k] +Qk=0dk w[n k]The corresponding system function is given by:H(z) =D(z)A(z)=Qk=0 dk zk1 +Pk=1 ak zkStochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 55/55THEUN IV ERSITYOFEDI N BURGHPole-Zero Frequency ResponseThe pole-zero model can be written asH(z) =D(z)A(z)= d0Qk=1(1 zk z1)Pk=1 (1 pk z1)Therefore, its frequency response is given by:H(ej) = d0Qk=1(1 zk ej)Pk=1 (1 pk ej)Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 55/55THEUN IV ERSITYOFEDI N BURGHPole-Zero Frequency ResponseThe pole-zero model can be written asH(z) =D(z)A(z)= d0Qk=1(1 zk z1)Pk=1 (1 pk z1)Therefore, its frequency response is given by:H(ej) = d0Qk=1(1 zk ej)Pk=1 (1 pk ej)The PSD of the output of a pole-zero filter is given by:Pxx(ej) = 2wH(ej)2= G2Qk=11 zk ej2Pk=1 |1 pk ej|2where G = w d0 is the overall gain of the system.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 55/55THEUN IV ERSITYOFEDI N BURGHPole-Zero Frequency Response0 0.2 0.4 0.6 0.8 101234567 / |H(ej)|PoleZero Model Magnitude Frequency Response 0.2 0.4 0.6 0.8 1302106024090270120300150330180 0Pole and Zero PositionsRe(z)Im(z)PolesZerosThe frequency response and position of the poles and zeros in anpole-zero system.Stochastic ProcessesPower Spectral DensityLinear Systems TheoryLinear Signal ModelsAbstractThe Ubiquitous WGNSequenceFiltration of WGNNonparametric andparametric modelsParametric Pole-Zero SignalModelsTypes of pole-zero modelsAll-pole ModelsFrequency Response of anAll-Pole Filter Impulse Response of anAll-Pole FilterAll-Pole Modelling andLinear PredictionAutoregressive ProcessesAll-Zero modelsFrequency Response of anAll-Zero FilterMoving-average processesPole-Zero ModelsPole-Zero FrequencyResponse- p. 55/55THEUN IV ERSITYOFEDI N BURGHPole-Zero Frequency Response0 0.2 0.4 0.6 0.8 180604020020 / 10 log 10 |Pxx(ej)|PoleZero Model Power SpectrumPower spectral response of an pole-zero model.Stochastic ProcessesDefinition of a Stochastic ProcessDefinition of a Stochastic ProcessDefinition of a Stochastic ProcessDefinition of a Stochastic ProcessDefinition of a Stochastic ProcessDefinition of a Stochastic ProcessInterpretation of SequencesInterpretation of SequencesInterpretation of SequencesInterpretation of SequencesPredictable ProcessesPredictable ProcessesPredictable ProcessesDescription using pdfDescription using pdfSecond-order Statistical DescriptionSecond-order Statistical DescriptionSecond-order Statistical DescriptionSecond-order Statistical DescriptionExample of calculating autocorrelationsExample of calculating autocorrelationsExample of calculating autocorrelationsTypes of Stochastic ProcessesTypes of Stochastic ProcessesTypes of Stochastic ProcessesTypes of Stochastic ProcessesTypes of Stochastic ProcessesStationary ProcessesOrder-N and strict-sense stationarityOrder-N and strict-sense stationarityWide-sense stationarityWide-sense stationarityWide-sense stationarityWide-sense cyclo-stationarityWide-sense cyclo-stationarityQuasi-stationarityQuasi-stationarityQuasi-stationarityQuasi-stationarityWSS PropertiesWSS PropertiesWSS PropertiesWSS PropertiesWSS PropertiesEstimating statistical propertiesEstimating statistical propertiesEstimating statistical propertiesEstimating statistical propertiesEstimating statistical propertiesEnsemble and Time-AveragesEnsemble and Time-AveragesEnsemble and Time-AveragesErgodicityErgodicityJoint Signal StatisticsJoint Signal StatisticsTypes of Joint Stochastic ProcessesTypes of Joint Stochastic ProcessesTypes of Joint Stochastic ProcessesCorrelation MatricesCorrelation MatricesCorrelation MatricesCorrelation MatricesCorrelation MatricesMarkov ProcessesMarkov ProcessesMarkov ProcessesMarkov ProcessesMarkov ProcessesMarkov ProcessesMarkov ProcessesPower Spectral DensityIntroductionIntroductionIntroductionIntroductionIntroductionIntroductionIntroductionThe PSDThe PSDProperties of the PSDProperties of the PSDProperties of the PSDProperties of the PSDProperties of the PSDProperties of the PSDGeneral form of the PSDGeneral form of the PSDGeneral form of the PSDGeneral form of the PSDGeneral form of the PSDGeneral form of the PSDThe CPSDThe CPSDThe CPSDThe CPSDComplex Spectral Density FunctionsComplex Spectral Density FunctionsComplex Spectral Density FunctionsComplex Spectral Density FunctionsComplex Spectral Density FunctionsLinear Systems TheorySystems with Stochastic InputsSystems with Stochastic InputsSystems with Stochastic InputsLTI Systems with Stationary InputsLTI Systems with Stationary InputsLTI Systems with Stationary InputsInput-output Statistics of a LTI SystemInput-output Statistics of a LTI SystemInput-output Statistics of a LTI SystemInput-output Statistics of a LTI SystemSystem identificationLTV Systems with Nonstationary InputsLTV Systems with Nonstationary InputsDifference EquationDifference EquationDifference EquationDifference EquationDifference EquationFrequency-Domain Analysis of LTI systemsFrequency-Domain Analysis of LTI systemsLinear Signal ModelsAbstractAbstractAbstractAbstractAbstractAbstractAbstractThe Ubiquitous WGN SequenceFiltration of WGNFiltration of WGNFiltration of WGNFiltration of WGNNonparametric and parametric modelsNonparametric and parametric modelsParametric Pole-Zero Signal ModelsParametric Pole-Zero Signal ModelsTypes of pole-zero modelsTypes of pole-zero modelsTypes of pole-zero modelsTypes of pole-zero modelsAll-pole ModelsAll-pole ModelsFrequency Response of an All-Pole FilterFrequency Response of an All-Pole FilterFrequency Response of an All-Pole FilterFrequency Response of an All-Pole FilterFrequency Response of an All-Pole FilterFrequency Response of an All-Pole FilterImpulse Response of an All-Pole FilterImpulse Response of an All-Pole FilterAll-Pole Modelling and Linear PredictionAll-Pole Modelling and Linear PredictionAll-Pole Modelling and Linear PredictionAll-Pole Modelling and Linear PredictionAll-Pole Modelling and Linear PredictionAutoregressive ProcessesAutoregressive ProcessesAutoregressive ProcessesAutoregressive ProcessesAutoregressive ProcessesAutoregressive ProcessesAutoregressive ProcessesAll-Zero modelsAll-Zero modelsFrequency Response of an All-Zero FilterFrequency Response of an All-Zero FilterFrequency Response of an All-Zero FilterFrequency Response of an All-Zero FilterFrequency Response of an All-Zero FilterFrequency Response of an All-Zero FilterMoving-average processesMoving-average processesMoving-average processesPole-Zero ModelsPole-Zero Frequency ResponsePole-Zero Frequency ResponsePole-Zero Frequency ResponsePole-Zero Frequency Response

Recommended

View more >