technical note - autoregressive model

8/13/2019 Technical Note - Autoregressive Model

1/12

TechnicalNoteAutoregressiveModel 1 SpiderFinancialCorp,2014

Technical Note: AutoRegressive Model

Weoriginallycomposedthesetechnicalnotesaftersittinginonatimeseriesanalysisclass.Overthe

years,wevemaintainedthesenotesandaddednewinsights,empiricalobservationsandintuitions

acquired.Weoftengobacktothesenotesforresolvingdevelopmentissuesand/ortoproperlyaddress

aproductsupportmatter.

Inthispaper,wellgooveranothersimple,yetfundamental,econometricmodel:theautoregressive

model.Makesureyouhavelookedoverourpriorpaperonthemovingaveragemodel,aswebuildon

manyoftheconceptspresentedinthatpaper.

ThismodelservesasacornerstoneforanyseriousapplicationofARMA/ARIMAmodels.

Background

Theautoregressivemodeloforder p(i.e. ( )AR p )isdefinedasfollows:

1 1 2 2 2...

~ i.i.d ~ (0,1)

t o t t p t t

t t

t

x x x x a

a

N

Where

ta is

the

innovations

or

shocks

for

our

process

istheconditionalstandarddeviation(akavolatility)Essentially,the ( )AR p ismerelyamultiplelinearregressionmodelwheretheindependent

(explanatory)variablesarethelaggededitionsoftheoutput(i.e.1 2, ,...,t t t px x x ).Keepinmindthat

1 2, ,...,t t t px x x maybehighlycorrelatedwitheachother.

Whydoweneedanothermodel?

First,wecanthinkofanARmodelasaspecial(i.e.restricted)representationofaMA( ) process.Lets

considerthefollowingstationaryAR(1)process:

1 1

1 1 1 1

1 1(1 )( )

t o t t

t o t t

t o t

x x a

x x a

L x a

Now,bysubtractingthelongrunmeanfromtheresponsevariable(tx ),theprocessnowhaszerolong

run(unconditional/marginal)mean.


2/12


1

1

1

0

1

1

o

o

Next,theprocesscanbefurthersimplifiedasfollows:

1 1

1

(1 )( ) (1 )

1

t t t

tt

L x L z a

az

L

Forastationaryprocess,the 1 1

2 2 3 31 1 1 1

1

(1 ... ...)1

N Ntt taz L L L L a

L

Insum,usingtheAR(1)model,weareabletorepresentthis MA( ) model usingasmallerstorage

requirement.

WecangeneralizetheprocedureforastationaryAR(p)model,andassumingan MA( ) representation

exists,theMAcoefficientsvaluesaresolelydeterminedbytheARcoefficientvalues:

1 1 2 2

1 2 1 1 1 2 2 2

2

1 2 1 2

...

... ...(1 ... )( ) ..

t o t t p t p t

t o p t t p t p p t

p

p t o p t t

x x x x a

x x x x aL L L x a a

Onceagain,bydesign,thelongrunmeanoftherevisedmodeliszero.

1 2

1 2

1

.. 0

1 ...

1

o p

o

p

p

i

i

Hence,theprocesscanberepresentedasfollows:

2

1 2

2

1 2 1 2

(1 ... )

( )1 ... (1 L)(1 L)..(1 L)

p

p t t

t tt t p

p p

L L L z a

a ax z

L L L


3/12


Byhaving 1, {1,2,.., }i i p ,wecanusethepartialfractiondecompositionandthegeometric

seriesrepresentation;wethenconstructthealgebraicequivalentofthe MA( ) representation.

Hint:Bynow,thisformulationlooksenoughlikewhatwehavedoneearlierintheMAtechnicalnote,

sinceweinvertedafiniteorderMAprocessintoanequivalentrepresentationof ( )AR .

Thekeypointisbeingabletoconvertastationary,finiteorderARprocessintoanalgebraically

equivalent MA( ) representation.Thispropertyisreferredtoascausality.

Causality

Definition:Alinearprocess{ }tX iscausal(strictly,acausalfunctionof{ }ta )ifthereisanequivalent

MA( ) representation.

0

( ) it t i t i

X L a L a

Where:

1

i

i

Causalityisapropertyofboth{ }tX and{ }ta .

Inplainwords,thevalueof{ }tX issolelydependentonthepastvaluesof{ }ta .

IMPORTANT:AnAR(p)processiscausal(withrespectto{ }ta )ifandonlyifthecharacteristicsroots(i.e.

1

i)falloutsidetheunitcircle(i.e.

11 1i

i

).

Letsconsiderthefollowingexample:

1

(1 )( ) (1 ) z

1

z z

t t t

t t t

L x L a

a

Now,letsreorganizethetermsinthismodel:

1

"

1

1z (z )

z z 1

t t t

t t t

a

a


4/12


" " 2 " "

2 2 1 2 2 1

3 2 " " "

3 3 2 1

1 " 2 " " "

1 2 1

" " 2 " "1 2 3 1

z ( z ) z

z z

z z ...

z ... ...

t t t t t t t

t t t t t

N N N

t t N t N t N t t

Nt t t t t N

a a a a

a a a

a a a a

a a a a

Theprocessaboveisnoncausal,asitsvaluesdependonfuturevaluesof"

{ }ta observations. However,it

isalsostationary.

Goingforward,foranAR(andARMA)process,stationarityisnotsufficientbyitself;theprocessmustbe

causalaswell.Forallourfuturediscussionsandapplication,weshallonlyconsiderstationarycausal

processes.

StabilitySimilartowhatwedidinthemovingaveragemodelpaper,wewillnowexaminethelongrunmarginal

(unconditional)meanandvariance.

(1) Letsassumethelongrunmean( )exists,and:1[ ] [ ] ... [ ]t t t pE x E x E x

Now,subtractthelongrunmeanfromalloutputvariables:

1 1 1 2 2 2

1 1 2 2

1 2

( ) ( ) ... ( )

( ) ( ) ( ) ... ( )

+ (1 ... )

t o t t p t p p t

t t t p t p t

o p

x x x x a

x x x x a

Taketheexpectationfrombothsides:

1 1 2 2

1 2

1 2

1 2

1

[ ] [ ( ) ( ) ... ( ) ]

+ (1 ... )

0 (1 ... )

1 ...

1

t t t p t p t

o p

o p

o

p

p

i

i

E x E x x x a

Insum,forthelongrunmeantoexist,thesumofvaluesoftheARcoefficientscantbeequalto

one.


5/12


(2) ToexaminethelongrunvarianceofanARprocess,wellusetheequivalent MA( ) representationandexamineitslongrunvariance.

1 1 2 2 3 3

2

1 2

2

1 2

...

(1 ... )

1 ...

t t t t t p t p t

p

p t t

tt p

p

x y y y y y a

L L L y a

ay

L L L

Usingpartialfractiondecomposition:

1 2

1 2

...1 1 1

p

t t

p

cc cy a

L L L

ForastableMAprocess,allcharacteristicsroots(i.e. 1

i)mustfalloutsidetheunitcircle(i.e.

1i ):

2 2 2 2

1 2 1 1 2 2 1 1 2 2( ... ) ( ... ) L ( ... ) L ...t p p p p p t y c c c c c c c c c a

Next,letsexaminetheconvergencepropertyoftheMArepresentation:

1 1 2 2lim ... 0k k k

p pk

c c c

Finally,thelongrunvarianceofaninfiniteMAprocessexistsifthesumofitssquared

coefficientsisfinite.

2 2

1 2 1 1 2 2

2 2

1 1 2 2

2 2

1 1 2 2

1 1 1

Var[ ] (1 ( ... ) ( ... ) ...

+ ( ... ) ...)

( ... ) ( )

T k p p pk

k k k

p p

pi i i i

p p j j

i i j

x c c c c c c

c c c

c c c c

Furthermore,fortheAR(p)processtobecausal,thesumofabsolutecoefficientvaluesisfinite

aswell.

1 1 1

pi

k j j

k i j

c


6/12


Example:AR(1)

2 2

22 4 6 2

2

(1 )

(1 ...)

1

Var[ ] (1 ...)1

t t

tt t

t

L y a

ay L L a

L

y

Assumingallcharacteristicroots(1

i)falloutsidetheunitcircle,theAR(p)processcanbeviewedasa

weightedsumofpstableMAprocesses,soafinitelongrunvariancemustexit.

ImpulseResponseFunction(IRF)

Earlier,weusedAR(p)characteristicsrootsandpartialfractiondecompositiontoderivetheequivalent

ofan

infinite

order

moving

average

representation.

Alternatively,

we

can

compute

the

impulse

response

function(IRF)andfindtheMAcoefficientsvalues.

Theimpulseresponsefunctiondescribesthemodeloutputtriggeredbyasingleshockattimet.

0 1

1 1t

ta

t

1 1

2 1 1 1

3 1 2 2 1

4 1 3 2 2 3 1

5 1 4 2 3 3 2 4 1

1 1 2 1 3 2 1

2 1 1 2 3 1 2

1 1 2 2 3 3

1

...

...

...

...

...

p p p p p

p p p p p

p k p k p k p k p k

y a

y y

y y yy y y y

y y y y y

y y y y y

y y y y y

y y y y y

Theprocedureaboveisrelativelysimple(computationally)toperform,andcanbecarriedonforany

arbitraryorder(i.e.k).

Note:Recallthepartialfractiondecompositionwedidearlier:1 2

1 2

...1 1 1

p

t t

p

cc cy a

L L L


7/12


WederivedthevaluesfortheMAcoefficientsasfollows:2 2 2 2

1 2 1 1 2 2 1 1 2 2( ... ) ( ... ) L ( ... ) L ...t p p p p p t y c c c c c c c c c a

Inprinciple,

the

IRF

values

must

match

the

MA

coefficients

values.

So

we

can

conclude:

(1) Thesumofdenominators(i.e.ic )ofthepartialfractionsequalstoone(i.e. 1

1

1p

i

i

c y

).

(2) Theweightedsumofthecharacteristicsrootsequalsto1 (i.e. 2 1

1

p

i i

i

c y

).

(3) Theweightedsumofthesquaredcharacteristicsrootsequalsto 21 2 (i.e.

2 2

3 1 21

p

i ii

c y ).

Forecasting

Givenaninputdatasample 1 2{ , ,..., }Tx x x ,wecancalculatevaluesofthemovingaverageprocessfor

future(i.e.outofsample)valuesasfollows:

1 1 2 2 ...T T T p T p T y y y y a

1 1 2 1 1

2 1 1 2 2

2

1 2 1 2 3 1 1 1 2 2

[ ] ...

[ ] [ ] ...

= ( ) ( ) ... ( )

T T T p T p

T T T p T p

T T p p T p p T p

E y y y y

E y E y y y

y y y y

Wecancarrythiscalculationtoanynumberofstepswewish.

Next,fortheforecasterror:

2

1 1 2 1 1 1

2 2

2 1 1 2 2 2 1

3 1 2 2 1 3 3

1 1 1 2 2 2

Var[ ] Var[ ... ]

Var[ ] Var[ ... ] (1 )

Var[ ] Var[ ... ]

Var[ ( ...

T T T p T p T

T T T p T p T

T T T p T p T

T T p T p T

y y y y a

y y y y a

y y y y a

y y y a

2 1 3 3

2 2 2 2 2

1 2 1 1 2 3 1 1 2

) ... ]

Var[( ) .... .... ] (1 ( ) )

T p T p T

T T T

y y a

y a a

Asthenumberofstepsincrease,theformulasbecomemorecumbersome.Alternately,wecanusethe

MA( ) equivalentrepresentationandcomputetheforecasterror.


8/12


2

1 2IRF={ } (1 ...)t t tz z L L a

Andtheforecasterrorisexpressedasfollows:

21

2 2

2 1

2 2 2

3 1 2

2 2 2 2

1 2 1

2 2 2

1 2

Var[y ]

Var[y ] (1 )

Var[y ] (1 )

....

Var[y ] (1 ... )

....

Var[y ] (1 ...)

T

T

T

T k k

T kk

Note:The

conditional

variance

grows

cumulatively

over

an

infinite

number

of

steps

to

reach

its

long

run

(unconditional)variance.

CorrelogramWhatdotheautoregressive(AR)correlogramplotslooklike?HowcanweidentifyanARprocess(and

itsorder)usingonlyACForPACFplots?

First,letsexaminetheACFforanARprocess:

ACF(k)

k

ko

Where:

2

[( )( )] (covariance for lag j)

[( ) ] (long-run variance)

j t t j

o t

E x x

E x

Letsfirstcomputetheautocovariancefunctionj .

1 1 1

1 1 1 2 2 1 1 2 1 3 2 1

2 1 3 2 1 1

[( )( )] [ ]

[( .. ) ] ...

(1 ) ...

t t t t

t t p t p t t o p p

p p o

E x x E z z

E z z z a z

2 1 1 2 2 2 1 3 1 2 4 2 2

1 3 1 4 2 5 3 2 2

[( .. ) ] ( ) ...

( ) ( 1) ...

t t p t p t t o p p

p p o

E z z z a z


9/12


Next,forthe3rdlagcovariance;

3 1 1 2 2 3

3 1 2 2 1 3 4 1 5 2 6 3 2

2 4 1 1 5 2 6 3 7 4 2 3

[( .. ) ]

...

( ) ( ) ( 1) ...

t t p t p t t

o p p

p p o

E z z z a z

Insum,foranAR(p)process,weneedtoconstructandsolvep1linearsystemstocomputethevalues

ofthefirstp1autocovariances.

2 3 4 5 6 1

1 3 4 5 6 7

2 4 1 5 6 7 8

3 5 2 6 1 7 8 9

4 6 3 7 2 8 1 9 10

3 1 4 5 4 5

1 .

( ) 1 . 0

( ) ( ) 1 . 0 0

( ) ( ) ( ) 1 . 0 0

( ) ( ) ( ) ( ) 1 . 0 0. . . . . . . .

( ) ( ) .

p p

p

p p p p p p p

1 1

2 2

3 3

4 4

5 5

2 2

2 3 4 5 6 1 1 1

. .

0 0

( ) . 1

o

p p

p p p p p p p p

Theautocovarianceforlagsgreaterthanp1iscomputediterativelyasfollows:

1 1 2 2 1 1

1 1 2 1 1 2 1

2 1 1 2 1 3 2

1 1 2 2 1 1

...

...

...

...

...

p p p p p o

p p p p p

p p p p p

p k p k p k p k p k

Example:ForanAR(5)process,thelinearsystemofequationsoftheautocovariancefunctionsis

expressedbelow:

2 3 4 5 1 1

1 3 4 5 2 2

2 4 1 5 6 3 3

3 5 2 1 4 4

1

( ) 1 0

( ) ( ) 1 0

( ) 1

o

Q:whatdotheylooklikeintheACFplot?

Duetothecausalityeffect,ACFvaluesofatrueARprocessdontdroptozeroatanylagnumber,butrathertailexponentially.

ThispropertyhelpsustoqualitativelyidentifytheAR/ARMA(vs.MA)processintheACFplot. Determiningtheactualorder(i.e.p)oftheunderlyingARprocessis,inmostcases,difficult.


10/12


Example:LetsconsidertheAR(1)process:1t t tz z a

1 1

2

2 2 1

3

3 2

1

[ ][ ]

...

t t o

t t o

o

k

k k o

E z z

E z z

TheACFforanAR(1)processcanbeexpressedasfollows:

ACF(k) kk

o

TheACFvaluesdontdroptozeroatanylagnumber,butratherdeclineexponentially.

Q:WhataboutahigherorderARprocess?

TheACFplotcangetincreasinglymorecomplex,butitwillalwaystailexponentially.Thisisduetothe

modelscausalproperty.WecantellthedifferencebetweenanMAprocessandanAR/ARMAprocess

bythisqualitativedifference.

WeneedadifferenttoolorplottohelpidentifytheexactorderoftheARprocessanditsorder:aplot

thatdropstozeroafterthepthlagswhenthetruemodelisAR(p).Thistoolorplotisthepartialauto

correlationplot(PACF).

Partialautocorrelationfunction(PACF)

Thepartialautocorrelationfunction(PACF)isinterpretedasthecorrelationbetweentx and t hx ,

wherethelineardependencyoftheinterveninglags( 1 2 1, ,...,t t t hx x x )hasbeenremoved.

1 2 1PACF( ) ( , | , ,..., )t t h t t t hh Corr x x x x x

Notethatthisisalsohowtheparametersofamultiplelinearregression(MLR)modelsareinterpreted.

Example:

2

1

2

1 2

t o

t o

x t

x t t


11/12


Inthefirstmodel, 1 isinterpretedasthelineardependencybetween2

t andtx .Inthesecondmodel,

the2 isinterpretedasthelineardependencybetween

2t and

tx ,butwiththedependencybetween t

andtx alreadyaccountedfor.

Insum,thePACFhasaverysimilarinterpretationasthecoefficientsinthemultipleregressionsituations

andthePACFvaluesareestimatedusingthosecoefficientvalues.

(1) Constructaseriesofregressionmodelsandestimatetheparametersvalues:0,1 1,1 1

0,2 1,2 1 2,2 2

0,3 1,3 1 2,3 2 3,3 3

0,4 1,4 1 2,4 2 3,4 3 4,4 4

0, 1, 1 2, 2 3, 3 ,

...

...

t t t

t t t t

t t t t t

t t t t t t

t k k t k t k t k k t k t

x x a

x x x a

x x x x a

x x x x x a

x x x x x a

(2) ThePACF(k)isestimatedby,k k .

Notes:

(1) ToestimatethePACFofthefirstklags,wedneedtosolvekregressionmodels,whichcanbeslowforlargerdatasets.Anumberofalgorithms(e.g.DurbinLevensonalgorithmandYule

Walkerestimations)canbeemployedtoexpeditethecalculations.

(2) ThePACFcanbecalculatedfromthesampleautocovariance.Forexample,toestimatethePACF(2),wesolvethefollowingsystem:

1,21 1

2,21 2

o

o

ForPACF(3),wesolvethefollowingsystem:

1 2 1,3 1

1 1 2,3 2

2 1 3,3 3

o

o

o

UsingtheDurbinLevensonalgorithmimprovesthecalculationspeeddramaticallybyreusingprior

calculationstoestimatecurrentones.

[( )( )]j t t jE x x


12/12


Bydefinition,theautocovarianceoflagorderzero(o

)istheunconditional(marginal)variance.

Bydesign,foratrueAR(p)process,thecorrespondingPACFplotdropstozeroafterplags.Ontheother

hand,theACFplottails(declines)exponentially.

UsingonlythePACFplot,IshouldbeabletoconstructanARmodelforanyprocess,right?No.

ThePACFplotmainlyexamineswhethertheunderlyingprocessisatrueARprocessandidentifiesthe

orderofthemodel.

ConclusionTorecap,inthispaper,welaidthefoundationforaslightlymorecomplexmodel:theautoregressive

model(AR).First,wepresentedtheARprocessasarestrictedformofaninfiniteorderMAprocess.

Next,armedwithafewmathematicaltricks(i.e.IRF,partialfractiondecompositionandgeometric

series),wetackledmanymorecomplexcharacteristicsofthisprocess(e.g.forecasting,longrun

variance,etc.)byrepresentingitasanMAprocess.

Lateron,weintroducedanewconcept:Causality.Aprocessisdefinedascausalifandonlyifitsvalues

{ }t

X aredependentontheprocessspastshocks/innovations 1 2{ , , ,...}t t ta a a .Weshowedthat

stationarityisnotasufficientconditionforourmodels;theymustbecausalaswell.

Finally,wedelvedintoARprocessidentificationusingcorrelogram(i.e.ACFandPACF)plots.Weshowed

thattheACFofanARprocessdoesnotdroptozero,butrathertailsexponentiallyinallcases.

Furthermore,welookedintoPACFplotsandoutlinedthatfactthatPACF,bydesign,dropstozeroafter

plags

for

atrue

AR

process.

Aswegoontodiscussmoreadvancedmodelsinfuturetechnicalnotes,wewilloftenrefertotheMA

andARprocessesandthematerialpresentedhere.

References Hamilton, J .D.; Time Series Analysis , Princeton University Press (1994), ISBN 0-691-04289-6 D. S.G. Pollock,; Handbook of Time Series Analysis, Signal Processing, and Dynamics , Academic Press (1999),

ISBN: 0125609906

Box, Jenkins and Reisel; Time Series Analysis: Forecasting and Control , John Wiley & SONS. (2008) 4thedition, ISBN:0470272848

technical note - autoregressive model

Documents