woldpapulis.pdf

7/28/2019 woldpapulis.pdf

1/6

IEEE TRANSACTIONSONACOUSTICS, SPEECH,A NDS IGNALP R OC E S S ING, VOL. ASSP-33, NO . 4, AUGUST 1985

Predictable ProcessesandWold's Decomposition:A ReviewA. PAPOULIS

Abstmet-The concepts of predictability and band-limitedness arereexamined and a simple prooff Wold's decomposition s presented inthe context of mean-square estimation.

TI. PREDICTABLEN D BAND-LIMITEDROCESSESHE purpose of this paper is to clarify a number ofconceptsrelated o predictability and band-limited-ness; and to give a simple proof of Wold's decompositionin the context of linear mean-square (MS) estimation. Thepaper is essentially tutorial and, unlike most other treat-ments of this topic, mainly in the mathematical literature,the development is phrased in a language familiar to mostreaders of these transactions.

Considera eal,discrete-time, wide-sense stationaryprocess x [ n ] with autocorrelation

R [ m ] =E ( x [ n +m ] x [ n ] }and power spectrum

m

Clearly, S(2") is a periodic function with Fourier seriescoefficients R [ m ] .Hence,

E(x2[n ] } =R[O] =-L S(d")dw. (2)27r -7rIf x [ n ] is the input to a real, stable, linear system, then

the resulting outputm

y [ n ] = C h g [ n - k]k = - mis a wide-sense stationary process with power spectrum1 1 1

Sy(z)= S(z) H(z) w z - 9 (3)where

mH(z ) = hg-"n = - m

is the system function. From (3 ) and ( 2 ) it follows thatManu script received January 31,1984; revised November 8,1984. Thiswork was supported by the Joint Services Technical Advising Com mitteeunder Contract F4620-82-C-0084.The author is with the Polytechnic Institute of New York, Farmingdale,NY 11735.

933

E ( y 2 [ n ] )=RJO] =2- T S(2") (H(e'")(io . (4)2a -7rBand-Limited Processes

We shall say that a process x [ n ] is band limited (BL) ifS(e'") =0 w E D. (5)

where D s a set consisting of one ormore intervals (a setof positive measure).A n extreme case of a BL process is a process whosespectrum consists of lines only:

S(d")=c a16(w - W j ) . (6)I

In this case, the complement D of D s a countable set ofpoints wi.

We show next that the values of a BL process x [ n ] forn from - 0 to 03 are linearly dependent. For this purpose,we construct a continuous bounded functionf(w) such thatf ( w ) = 0 E D. (7)

f ( w ) S(e'")=0. (8)Hence, [see ( 5 ) ]

We, next, expand f(w into a Fourier series in the in-terval (- 7r, a)m

f(w> = C cne-Jnw. (9)The functionf(w) specifies a linear system with delta re-sponse c,. We maintain that, if x [ n ] is the input to thissystem, the resulting out y [ n ] is identically zero'

n=-m

co

y [ n ] = ck x [ n - k ] EE 0. (10)k = -mIndeed, as we see from (4) nd (8) ,

E { y 2 [ n ] )=- 7rS(d") F(w)l2 dw =0 (11)27r -7r,and (10) results.Conversely, if the values of a process x [ n ] are linearlydependent, i.e., if there exists a set of constants ck suchthat the sum y [ n ] in (10) is zero, then the process x [ n ] isBL.To prove the above, we form a periodic functionf(o) as

'All identities in this paper involving random variables will be in inter-preted in the MS sense.

0096-3518/85/0800-0933$01.00O 1985 IEEE


2/6

934 IEEE TRANSACTIONSN ACOUSTICS, SPEECH,NDIGNALROCESSING, VOL. ASSP-33,O. 4, AUGUST 19

in ( 9 ) with Fourier series coefficients ck. This function isdifferent from zero in a setD of positive measure becausethe constants c k are not all zero. Furthermore, the processy [ n ] is the output of a system with input x [ n ] and systemfunction f(w). And since y [ n ] = 0 by assumption, it fol-lows from (11) that

S ( P ) f ( w ) =0almost everywhere. This is possible only if S(d") = 0 inD.

We have, thus, shown that the values of a process x [ n ]are linearly dependent iff this process is BL.

We can assume, introducing a shift if necessary, thatco # 0. It then follows from (10)that

Thus, the present value x [ n ] of a BL process can be ex-pressed in terms of its past and future values. This rep-resentation is not, of course, unique because the functionf (w) is arbitrary subject only to the condition (8). We shallpresently show that a BL process can be approximatedarbitrarily closely by a sum involving only its past values.Furthermore, if S(d") consists of lines only as in (6), thenx [ n ] equals such a sum.Regular Processes

We shall say that a process x [ n ] is regular if

where L(z) is a function analytic for ( z (>1m

It can be shown that a process x [ n ] is regular iff it Sat-isfies the Paley-Wiener condition [21, [315" (In S(e'")l d w


3/6

PAPOULIS:PREDICTABLEPROCESSES AN D WOLD'SDECOMPOSITION 93 5

1 - E(z) ELz) (25)specify a family of predictors of x [ n ] of the form (20).If $(e'") is not a sum of impulses as in (21), E ( r 2 [ n ] )cannot be zero. Weshow next, however, that it can bearbitrarily small if the process x [ n ] is BL.Dejini'tion 2: Suppose that in (17) the minimum of theMS error does not exist. In this case, we define the, MSprediction error as the greatest lower bound

/I m

(26)We shall say that a process x [ n ] is weakly p redictuble if

P = Oor, equivalently, if the difference

mX[n] - c ak X[ n - k]

k = lcan be made arbitrarily small.

We note hat, if the minimum P in 17) exists, thenP = P . Thus, if a process is predictable, then it is alsoweakly predictable.

A process x [ n ] will. be called unpredictable if it is notpredictable or weakly predictable.Theorem 2 : A process x [ n ] is weakly predictable iff itis band limited.Proofl(a) Suficiency-We mustshow that, if S(e'") = 0 forw E D, hen, given E >0, we can find a set of constantsak such that

/I m

For this purpose, we form continuous even function q (w )consisting of straight line segments as in Fig. 1. The hor-izontal low-level segments are in the complementD of theset D nd theirheight equals 26. The horizontal high-levelsegments are in a subset G of D and their height equalsA + 6 . The total length of the set G equals 8. Thus,

In the above, 6 is an arbitrary positive constant andA =61 2u m (29)

From the continuity of q (w) , it follows that (Wierstrasstheorem) we can find a function

mp ( w ) = C cne-jnon = - m

such that

- H

From this and

I-Fig. 1.

(28) it follows that6

0, it follows from the FejCr-Riess theo-

rem [ 3 ] that we can find a Hurwitz polynomialm

E(z) = bnz-"n = Osuch that

e-'p(w) = I E ( ~ ' " ) I ~where [see (48) and (32)]

b: = exp [&--?r

Thus, E(z ) isan error filter as in (18) with a, = -bn,hence [see ( i s ) ]

E { E ' [ ~ ] ) $ S(e'") IE(&")12d u--?r

< 1 S@") p ( 0 ) u


4/6

936 IEEERANSACTIONS ON ACOUSTIC S,PEECH, ANDIGNAL PROCESSING,OL .SSP-33,O. 4, AUGUST 19

11. INNOVATIONSND PREDICTION4 ] , [ 5 ]The predictor

mf [ n ] =c ak x [ n - k ]3 5 )k = I

of a process x [ n ] can be considered as the projection ofits present value on the space spanned by its past. To de-termine the coefficientsa k rwe can, therefore, 'use the pro-jection theorem: the M S prediction error is minimum ifthe error

[ n ]=x [ n ] - f[n]is orthogonal to the data. This yieldsE x[n] - a k x [ n - k] x [ n - m ] =0 m 2 1from which it follows that

m[( k = l > Im

R [ m ] = ak R[m - k] m 2 1 ( 3 6 )This is the discrete-time form of the Wiener-Hopf equa-tion.We note for later use that the error ~ [ n ]s white noisewith power spectrum

k = O

S,,(Z) =P . ( 3 7 )Indeed, ~ [ n k] depends linearly on x [ n - k] and itspast; hence [see ( 3 7 ) ] it is orthogonal to ~ [ n ]or k 1 1 .

To determine the coefficients a k in (3 .3 , we must solvethe system ( 3 6 ) . We show next that, if x [ n ] is a regularprocess, then ( 3 6 ) can be solved simply with the methodof innovations. This method is an extension of the Gram-Schmidt orthonormalization to the space spanned by x [ n ]and its past.Innovations

As we noted, a process x [ n ] is regular if its spectrumcan be factored as in ( 1 3 ) ,where L(z) s a function analyticfor Iz( 1 1 . Without loss of generality, we can assumealso that L ( z ) is minimum-phase, i.e., that it and its in-verse

are analytic for Iz 1 > 1 . Indeed, if zi re the zeros of L(z)outside the unit circle, then replacing all factors z - zi ofL(z)by the factors z zi - 1, we obtain a minimum-phasefunction satisfying (13) because

Thus, the functions L ( z ) and F(z) can be expanded intopower series

convergent for IzI > 1 . They specify, therefore, two linecausalsystems with delta responses I n and yn , respetively. The systemL(z)will be called innovationsfilter anthe system r ( z )whiteningfilter. Using as input to the sytem r(z) he process x [ n ] , we obtain as output anothprocess

mi[n] = yk x[n - k ] . ( 3k = O

This process will be called the innovations of x [ n ] .SinS( z ) =L(z)L(z- '1 ( 4

s&) = S(Z) r(Z-l)= 1 . (4it follows from ( 3 ) thatThus, i [n ] is unit-power white noise. As we see from Fi2 , if i [ n ] s the nput to the innovations filter L ( z ) , hen thresulting response equals x [ n ].Thus,

mx[n] = c l k i [n - k ] .4 2 )k = O

From ( 3 9 )and ( 4 2 ) t follows that the processes x [ n ] ani [n ] are linearly equivalent in the sense that each is linarly dependent on 'the other and its past. This shows ththe predictor%[n]of x [ n ] can be expressed in terms of tpast of i [ n ] .

We maintain, in fact, thatm

%[a]= c lk i [ n - k ] .4 3 )k = 1

Indeed, the resulting error equals [n ] =x [ n ] - [n] = lo i [ n ] . (4

This error is orthogonal to i [ n - ] fork I because i[is white noise. Hence (orthogonality principle), i [ n ] s thpredictor of x [ n ]

We have, thus, expressed f [ n ] in terms of the innovtions of x [ n ] . As we see from ( 4 3 ) and ( 4 2 ) , f [ n ] s thoutput of the system

&(z ) =L(z) - o ( 4with input i [ n ] .To complete the specification of % [ n ] ,wmust express i [n ] in terms of x [ n ] using the innovatiofilter r(z) Fig. 3 ( a ) ] .The resulting system [Fig. 3(b)]

m mL(Z)= C I,, z - ~ r(z)= C yn z - ~ is theWiener predictor of x [ n ] hat is, its response to x[n = O n = O ( 3 8 ) equals % i n ] .


5/6

PAPOULIS:PREDICTABLEPROCESSES AN D WOLD'S DECOMPOSITION 937

i----------1

(b)Fig. 3.

The Kolm ogor of-Szeg o Erro r FormulaFrom (44) it follows that

P =E ( E2 [ n ] }= 1;. (47)We shall use the above to express P directly in terms ofthe power spectrum S($") ofx[n]. To do so, we show,

first, thatIn =- lr In IL(e"")(2w. (48)2 n --r

Proof [6],[7]: Cleariy,

SI , In IL(d")(2w = 4 In [L( z )L(z-')l dz

111. WOLD'SDECOMPOSITION8]-[ll]Wold's decomposition theorem states that an arbitraryunpredictable process x [ n ] can be written as a sum

x[n] =x , [n ] +xp[n] ( 51)where XJn] is a regular process and xp[n] is a predictableprocess orthogonal to x,[n] .

To prove this theorem, we form the predictor 2[n] ofx [ n ] as in ( 3 5 ) , and the normalized prediction error (Fig.4)

where P is the MS error asP > O

by assumption. As we know [see ( 3 7 ) ] , he process { [ n ]is white noise andSf&) = 1. ( 5 3 )

We next form the MS estimate of x[n] in terms of 5[n]and its past. This estimate is a sumOD

x,[n] = wk 5 [n - k] (54)k = Owhen C is the unit circle.Furthermore,where the weights wk areuchs to minimize the MS value-of the error

xp[n]=x [ n ] - x , [ n ] . ( 5 5 )To prove (48), it suffices, therefore, to show that From the orthogonality principle it follows that wk must besuch that

(49)As we know, L ( z ) is minimum phase, hence, the func-

tion In L(z) is analytic for IzI 2 1. We can, therefore,replace in ( 4 9 ) the circle C with a circle whose radius isarbitrarily large (Cauchy's theorem). And since

U Z )- 0IZ I-+-the integral in (49) equals 27rj In (1,l. This yields (49).

Comparing.(47) with (48), we obtain

xP[n]I 5 [n - k] k 2 0. (56)Furthermore,

E [ a ] I [n - k], ~,[n- k], xp[n - k] k 2 1because x, [n] depends linearly on e [ n - k] and its past,E [n ] is white noise, and E [n] I x [ n - k] for k 2 1 (or-thogonality principle). Hence [see (5411,

E(x,[n +r n ] x , [ n ] } =0 all m . (57)Thus, the processes x, [n] and x,[n] are orthogonal and

E ( x 2 [ n ] )=E(x: [ i z ] ) +E(x;[nI}. ( 5 8 )The above leads to the conclusion that [see (53)]

This result is known as the Kolmogoroff-Szego MS error 4 E{x:[n]} 5 E(x2[n]} =R[O]


6/6

IEEE TRANSACTIONS ON ACOUSTICS,SPEECH,AN DSIGNALPROCESSING,VOL.ASSP-33,NO. 4, AUGUST 19

Therefore, the summ

W(Z)= c w, -n (60 )n= Oconverges for Iz I >1 and it defines a linear causal system.Clearly, if r[n] s the input to this system, then the result-ing output equals x , [ n ] . Hence, [see (53) and ( 3 ) ] , hepower spectrum of x , [n ] equals

msr(d")=I c w, e-jn"n = O 1. (61)We have, thus, shown that theM S estimate x,[n] of x [ n ]is terms of r [ n ]and its past is a regular process rthogonalto the error x p [ n ] .To complete the proof of the theorem,

we must show that x p [ n ] s a predictable process. We shallshow, in fact, that

mxp[n] = ak xp[n - k]62)k = l

where akare the coefficients of the predictorm

H(Z) = c a , z - (63 )n= 1

of x [ n ] .To prove ( 6 2 ) , we must show that, ifm

y [ n ] =xp[n]- ak xp[n - k] (64)k = 1then

E { y 2 [ n ] }= 0. (65 )Clearly, y [ n ] is the output of the error filter

E(z ) = 1 - C an -" = 1 - H(z )with input xp[n]=x [ n ] - x , [n ] . And since the responseof E ( z ) to x [ n ] equals ~ [ n ] ,e conclude that

m

n = 1

~ [ n l E [nl - r[nIwhere y , [n ] is the response of E(z ) to x , [ n ] . From this itfollows that the process y [ n ] is the output of the causalfilter (see Fig. 5 )

with input E [ n ] .Hence y [ n ] is linearly dependent on E [and its past. But y [ n ] is also orthogonal to ~ [ n ]nd its pa[see ( 6 4 ) and ( 5 6 ) ] ,hence y [ n ] =0 in the MS sense.

From the preceding discussion it follows that the spetrum S(e'") of an unpredictable process x [ n ] is a sum

S(e'") = Sr(e'") +SP(eiW) (6where S,.(d") s the spectrum of x,[n] and S,(d") is thspectrum of x p [ n ] .The first term is the sum in (61) anthe second term is a sum of impulses as in ( 2 1 ) .

We have thus shown that the processes x [ n ] and x phave the same predictors. However, whereas H ( z ) is thunique predictor of x [ n ] , he process x p [ n ]has many prdictors [see ( 2 5 ) ] . n fact, H ( z ) is not its simplest preditor. The minimum degree predictor of x p [ n ] s the polnomial

H,(Z) = 1 - T (1 - 2-' P i )I

where t P i are the zerosof E(z ) on the unit circle [see ( 2 4REFERENCES

A . Papoulis, Probability, Random Variables,an dStochasticPro-cesses , 2n dEd. NewYork: McGraw-Hill,1984.R . E . A. C. Paley and N. Wiener, "Fou rier transforms in the compldomain," Amer. Math. Soc. Coll., vol.19,1934.A . Papoulis, Signal Analys is. NewYork: M cGra w-H ill, 1977.T. Kailath, "An innovations approach to detection and estimation ory,'' Proc. IEEE, vol. 58, 1970.N. Wiener, Extrapolation, Interpolation and Smoothing of StationaTime Series. NewYork: MIT Press and Wiley,1950.G. Sz ego , "Orthogonal polynomials," Amer. Math. SOC . Col l . , 193A. Papoulis, "Maximum entropy and spectral estimation: A reviewIEEE Trans. Acoust. Speech, Signal Processin g, vol. ASSP-29, De1981.J. L . Doob, StochasticProcesses. NewYork:Wiley,1953.J. Lamperti, StochasticProcesses. NewYork: Sp ringe r, 1977.M . B. Priestley, Spectral Analysis and Time Series. New York: Aademic, 1982.S. Haykin, NonlinearMethods of Spectral Analysis. Berlin, Gmany: Springer, 1983.

woldpapulis.pdf

Documents