mixtures of nonlinear poisson autoregressions · 2 days ago · mixtures of nonlinear poisson...

Mixtures ofNonlinear PoissonAutoregressions

J. Rynkiewicz1 , Jointwork with P.Doukhan, K.

Fokianos

Introduction

The modelA Simple Mixture Model

On ergodicity andstationarity ofMINARCH(K , ∞)modelsWeakly dependance

Mixing for MS-INARCH(K ,L) models

Maximum LikelihoodInference forMS-INARCH(K , L)modelsThe log-likelihoodfunction

Consistency

Inference for knownnumber of regimes

Unknown number ofregimes

Application

Mixtures of Nonlinear Poisson Autoregressions

J. Rynkiewicz1, Joint work with P. Doukhan, K. Fokianos

1-Université de Paris I, Paris, France

Online Opening EcoDep Conference,September 2020

J. Rynkiewicz1 , Joint work with P. Doukhan, K. Fokianos Mixtures of Nonlinear Poisson Autoregressions



Fokianos

Introduction





Consistency



Application

Introduction

We consider models for count time series, which allow for the mean process tochange according to the values of an unobservable discrete random variable.

We introduce Markov switching non linear autoregressive Poisson modelsand study their properties by considering two cases :

1 For the mixture setup, we show that infinite order models are weaklydependent.

2 For the Markov switching case, we prove that finite order models are geometricβ-mixing.

We study the statistical properties of the maximum likelihood estimator forthe case of finite order autoregression of Markov switching models.

1 When the number of regimes is known.2 When the number of regimes is unknown, we show that a marginal likelihood

ratio test for testing the number of hidden regimes converges to a Gaussianprocess.

We apply the theoretical results to the weekly number of E.coli cases andcompare mixture models to models with interventions.




Fokianos

Introduction





Consistency



Application

Markov Switching Models for Count Time Series

We assume that (Zt )t∈N and (Yt )t∈N are two random sequences definedon a probability space (Ω,G,P) and taking values in a finite setK = 1, · · · ,K and in N, respectively.

Suppose that (Zt )t∈N is the unobserved "state sequence" which is atime-homogeneous Markov chain with transition probability matrixQ = (qi,j )1≤i,j≤K , i.e., for any integer t ∈N, and for any i, j ∈ K,

P(Zt+1 = j |Zt = i ) = qi,j . (1)

Moreover, p = (pi )i∈K will denote the initial probability distribution of theprocess (Zt )t∈N.

The observations (Yt )t∈N, conditionally on (Zt )t∈N and on the σ-field Ft ,generated by Ys, s ≤ t, are distributed according to a Poisson law.

P[Yt = y |Zt = k ,Ft−1] =exp(−λk ,t )λ

yk ,t

y !, y = 0,1,2, . . . , (2)




Fokianos

Introduction





Consistency



Application

A Simple Mixture Model

When (Zt ) is an iid sequence of random variables with P[Zt = k ] = pk fork = 1, . . . ,K , the marginal probability mass function (pmf) of (Yt ), giventhe past, is given by

P[Yt = y | Ft−1] =K

∑k=1

pkexp(−λk ,t )λ

yk ,t

y !, y = 0,1,2, . . . ,

We say that the conditional distribution of Yt , given the past, is a K -classmixture of Poisson distributions.

Elementary calculations show that

E[Yt | Ft−1] = ∑Kk=1 pk λk ,t ,

Var[Yt | Ft−1] = ∑Kk=1 pk λk ,t (1 + λk ,t )−

(∑K

k=1 pk λk ,t

)2.

Since ∑Kk=1 pk λ2

k ,t −(

∑Kk=1 pk λk ,t

)2= ∑K

k=1k 6=l

∑Kl=1 pk pl (λk ,t − λl,t )

2 > 0

we have shown that simple a Poisson mixture models takes into accountoverdispersion, which usually observed in count time series analysis.




Fokianos

Introduction





Consistency



Application

On ergodicity and stationarity of MINARCH(K , ∞) models

TheoremConsider the MINARCH(K , ∞) model. Assume that for any x and x′ inN∞ ×N∞ (with N = 0,1,2, . . .), there exist sequences (αkl )l≥1,k = 1, . . . ,K of non-negative real numbers such that

|fk (x)− fk (x′)| ≤∞

∑l=1

αkl |xl − x ′l | k = 1,2, . . . ,K .

Denote by Ak = ∑l αkl and let Bs = ∑Kk=1 pk As

k < 1. Then (i) If s = 1 thereexists a τ−weakly dependent strictly stationary process Yt , t ∈ Z whichbelongs to L1 and (ii) for s > 1 this solution belongs to Ls . Moreover

τ(r ) ≤ 21− B1

max1≤k≤K

fk (0) inf1≤p≤r

B

rp1 +

11− B1

∞

∑q=p+1

K

∑k=1

pk αkq

,

where (τ(r ))r∈N denotes the sequence of τ-dependence coefficients.




Fokianos

Introduction





Consistency



Application

Mixing for MS-INARCH(K , L) models

We can show that the MS-INARCH(K , ∞) is geometrically β-mixing understricter conditions.

Proposition

For L < ∞, consider the MS-INARCH(K , L) model. Assume that for any x andx′ in NL ×NL (with N = 0,1,2, . . .), there exist sequences (αkl )l≥1,k = 1, . . . ,K of non-negative real numbers such that

|fk (x)− fk (x′)| ≤L

∑l=1

αkl |xl − x ′l | k = 1,2, . . . ,K .

Suppose further that α = ∑Ll=1 max1≤k≤K αkl < 1. Then the Markov chain

Wt = (Yt , . . . ,Yt−L+1,Zt ) ∈NL ×K is geometrically β−mixing.




Fokianos

Introduction





Consistency



Application

Likelihood for MS-INARCH(K , L) models

We investigate the behavior of maximum likelihood estimator (MLE) forMS-INARCH(K ,L) models.

Assume that the observations Yt , t = 1, . . . ,n is a realization of astrictly stationary process Yt , t ∈ Z.Denote the vector of unknown parameters by ψ = (vec(Q),ψT

1 , . . . ,ψTK )

T

where Q is the unknown transition probability matrix Q = (qi,j )1≤i,j≤K .

We introduce the function

gk (y , y1, . . . , yL;ψk ) =exp(−f (y1, . . . , yL;ψk ))f y (y1, . . . , yL;ψk )

y !,

Suppose that we have observations y−L+1, · · · , yn from the (Yt )t∈N

sequence. Then, the conditional likelihood, given y−L+1, · · · , y0, is equalto

Ln(y1, · · · , yn |y−L+1, · · · , y0 ;ψ) =

∑Kz1=1 Ln(y1, · · · , yn |y−L+1, · · · , y0,Z1 = z1 ;ψ)Pψ(Z1 = z1 |y−L+1, · · · , y0 )

= ∑Kz1=1 · · ·∑K

zn=1 ∏nt=2 qzt−1,zt

∏nt=1 gk (yt , yt−1, . . . , yt−L;ψk )Pψ(Z1 = z1 |y−L+1, · · · , y0 ).




Fokianos

Introduction





Consistency



Application

Modified conditional likelihood

Multiplying the preceding equation by the density L(y−L+1, · · · , y0;ψ) of(Y−L+1, · · · ,Y0) yields the likelihood. In general, it is not feasible tocompute the stationary distributions of (Z1,Y−L+1, · · · ,Y0) ; thus we willwork with the modified conditional likelihood :

Ln(y1, · · · , yn |y−L+1, · · · , y0 ;ψ) =

∑Kz1=1 · · ·∑K

zn=1 πz1 ∏nt=2 qzt−1,zt ∏n

t=1 gk (yt , yt−1, . . . , yt−L;ψk ).

The vector πz1 of initial state probabilities may be chosen as an arbitrarystochastic vector with positive entries.

With a slight abuse of language, we call Maximum Likelihood Estimator(MLE) a maximum point of the modified likelihood, say ψn defined by

ψn = argmaxψ∈Ψ

Ln(y1, · · · , yn |y−L+1, · · · , y0 ;ψ).

It turns out that the modified likelihood can be calculated in linear time byemploying a recursive state filter.




Fokianos

Introduction





Consistency



Application

Assumptions for the consistency and asymptotic normalityof the conditional MLE

H-1 : The Markov chain (Yt ,Zt )t∈N admits a stationarysolution.

H-2 : The parameter vector ψ belongs to a compact setΨ ⊂ Rd , and the true parameter ψ0 belongs to the interior ofΨ.

H-3 The regression functions fk (·) are continuous and theysatisfy that fk (·) ≥ C1 for some constant C1 > 0 and for allk = 1, . . . ,K .

H-4 : Assume that Pψ = Pψ0 if and only if ψ = ψ0.

H-5 : For all (y1, . . . , yL) ∈NL, the functions ψk 7→ fk (·) aretwice continuously differentiable on Ψη .

H-6 : E(supψ∈Ψη

supk∈K ‖∇ψfk (·)‖2)< ∞ and

E(supψ∈Ψη

supk∈K ‖∇2ψfk (·)‖

)< ∞.




Fokianos

Introduction





Consistency



Application

Consistency

LemmaConsider the MS-INARCH(K , L) model. Then under assumptions ( H-1)–(H-3)it holds that, as n→ ∞,

1n

Ln(y1, . . . , yn |y−L+1, · · · , y0 ;ψ)a.s.−→ H(ψ),

where H(.) is the entropy function. We define the Kullback-Leibler divergenceby KL(ψ) = H(ψ0)−H(ψ).

TheoremConsider the MS-INARCH(K , L) model. Then under assumptions( H-1)–( H-3) it holds that, as n→ ∞,

ρ(ψn,Ψ0)a.s.−→ 0.




Fokianos

Introduction





Consistency



Application

Inference for known number of regimes

If the regressions functions fk (.) are identifiable, and the number ofregimes is known then we can easily show that the model MS-INARCH(K ,L) is identifiable and the MLE is consistent.

TheoremConsider the MS-INARCH(K , L) model and assume that (H-1)–(H-4) holdtrue. Then, as n→ ∞, limn→∞ ψn=ψ0, a.s.

Asymptotic normality of the MLE is implied by a central limit theorem forthe score function ∇ψ log Ln/

√n, and a locally uniform law of large

numbers for the observed Fisher information −∇2ψ log Ln/n in a

neighborhood of ψ0.

TheoremSuppose that (H-1)–(H-6) hold true and I(ψ0) is positive definite. Then, asn→ ∞

11√n∇ψLn(y1, . . . , yn |y−L+1, . . . , y0 ;ψ0) −→ N

(0, I(ψ0)

),

21n∇2

ψLn(y1, . . . , yn |y−L+1, . . . , y0 ;ψ0) −→ I(ψ0),

3 √n(

ψn −ψ0)−→ N

(0, I−1(ψ0)

).




Fokianos

Introduction





Consistency



Application

Unknown number of regimes

If the number of hidden regimes is unknown, then the MS-INARCH modelis not identifiable and therefore the likelihood ratio test statistic (LRT) hasa non-standard behavior.The following simple example illustrates this issue. Assume that the truedata generating process has only K 0 = 1 regime and the autoregressivefunction is linear with L = 1, i.e.

P(Yt = y |Yt−1 ) = exp(−(ψ00 + ψ0

1Yt−1)(ψ0

0 + ψ01Yt−1)

y

y !.

We fit a linear model with K = 2 regimes and transition matrix withelements qi,1 = p1 and qi,2 = 1− p1, for i = 1,2,

P(Yt = y |Yt−1 ) = p1 exp(−(ψ1,0 + ψ1,1Yt−1))(ψ1,0+ψ1,1Yt−1)

y

y ! +

(1− p1) exp(−(ψ2,0 + ψ2,1Yt−1))(ψ2,0+ψ2,1Yt−1)

y

y ! .

Any parameter vector ψ = (p1,ψ1,0,ψ1,1,ψ2,0,ψ2,1) withψ1,0 = ψ2,0 = ψ0

0,ψ1,1 = ψ2,1 = ψ01, p1 ∈ [0;1] or

ψ2,0 = ψ00,ψ2,1 = ψ0

1, (ψ1,0,ψ1,1) ∈ R2, p1 = 0 orψ1,0 = ψ0

0,ψ1,1 = ψ01, (ψ2,0,ψ2,1) ∈ R2, p1 = 1 satisfies the true

conditional pmf. This simple observation shows that for any parametricmodel, whose number of regimes is overestimated, has a non-invertibleFisher information matrix and it explains why the LRT has a non-standardasymptotic behavior.




Fokianos

Introduction





Consistency



Application

The marginal-conditional density function

we study the asymptotic distribution of the marginal LRT (MLRT), whenthe number of mixture regimes is overestimated.

Denote by g(y , y1, · · · , yL;ψ) the marginal-conditional density function ofy knowing y1, · · · , yL, i.e.

g(y , y1, · · · , yL;ψ) =K

∑k=1

pk (y1, . . . , yL;ψ)gk (y , y1, . . . , yL;ψk ), (3)

where pk (y1, . . . , yL;ψ) is the stationary law of conditional expectation ofthe prediction filter pk ,t knowing the immediate pastYt−1 = y1, · · · ,Yt−L = yL.

Moreover, we will denote by g0 the true marginal-conditional density :

g0(y , y1, · · · , yL) =K 0

∑k=1

pk (y1, · · · , yl ;ψ0)gk (y , y1, . . . , yL;ψ0k ),

for any parameter vector ψ0 ∈ Ψ0, realising the true density function ofthe observations.




Fokianos

Introduction





Consistency



Application

Some notations

for any parameter vector ψ0 ∈ Ψ0, realising the true density function ofthe observations.For η > 0, denote by Gη =

g ∈ G, ‖g − g0‖L2(µ) ≤ η

the set of

conditional densities in a η-neighborhood of the true density whereG = g(.;ψ),ψ ∈ Ψ.In a η-neighborhood of the true density, the extended set ofscore-functions Sη is defined by

Sη =

sg =

gg0 −1∥∥∥∥ g

g0 −1∥∥∥∥

L2(µ)

,g ∈ Gη

.

We define the limit-set of score functions D

D =

d ∈ L2(µ) | ∃ (gn) ∈ G, ‖ gn−g0

g0 ‖L2(µ) −−−→n→∞0,

‖d − sgn‖L2(µ) −−−→n→∞0.

Let

ln(ψ) =n

∑k=1

lng(yk , yk−1, · · · , yk−L)

be the marginal likelihood of y1, · · · , yn knowing, y0, · · · , y1−L.




Fokianos

Introduction





Consistency



Application

Assumptions

We assume the following :T-1 The process (Yt )t∈N is geometrically β-mixing.T-2 The parameter space Ψ is a compact set of Rd , and the

parameter space Ψ contains a neighborhood of theparameters defining the true conditional density g0. Moreoverall the entries of the transition matrix Q are supposed to bestrictly positives. Since Ψ is compact this means that all theentries of the transition matrix Q are bounded by below.

T-3 There exists η > 0 such that for all g ∈ G with

‖g − g0‖L2(µ) ≤ η,∥∥∥ g

g0 − 1∥∥∥

L2(µ)< ∞.

T-4 Assumptions (H-5) and (H-6) hold true. Note that (H-4) doesnot generally hold if the number of regimes is not known.

T-5 With the following notations :

l ′k :=∂lψk

∂ψk

(ψ0

k

), l ′′k :=

∂2lψk

∂ψk2

(ψ0

k

)we assume that for distinct (ψk )1≤k≤K(

lψk

)1≤k≤K 0 ,

(l ′k)

1≤k≤K 0 ,(l ′′k)

1≤k≤K 0

are linearly independent in the Hilbert space L2(µ).




Fokianos

Introduction





Consistency



Application

Asymptotic behavior

TheoremConsider the MS-INARCH(K , L) model and assume that (T-1)–(T-5) hold true.Then, the MLRT is represented by

λn = supd∈D

(max

1√n

n

∑i=2

d(Y1, · · · ,YL);0

)2

+ oP (1),

recalling that D denotes the limit-set of score functions. Hence, there exists acentered Gaussian process WS ,S ∈ F with continuous sample path andcovariance kernel P

(WS1

WS2

)= P (S1S2) such that

limn→∞

λn = supS∈F

(max(WS ,0))2 ,

where the index set F of limit derivatives depends on the compact set Ψ ofpossible parameters.

We can verify of assumptions (T-1)–(T-5) for the special case of linearMS-INARCH(K , 1) model.This theorem implies that asymptotic distribution is bounded in probabilitybut it depends on the true model parameters. Therefore, the theoremcannot be used to applications for obtaining critical values.




Fokianos

Introduction





Consistency



Application

estimating the true number of regimes (1)

For l ∈N+, denote

Gl = MS-INARCH(l,L) with l regimes and parameters set Ψl .

For some fixed M ∈N+ sufficiently large, we shall consider the followingclass of functions

GM =M⋃

l=1

Gl .

For every g ∈ GM define the number of regimes as

l(g) = min l ∈ 1, . . . ,M , g ∈ Gl .

Then, l0 = l(g0) denotes the number of regimes of the true model. An

estimate of the number of regimes, say l is defined as the integerl ∈ 1, . . . ,M which maximizes the following penalized criterion,

Tn (l) = supψ∈Ψl

ln (ψ)− αn (l) ,

where αn(·) is a suitably chosen penalization sequence.




Fokianos

Introduction





Consistency



Application

estimating the true number of regimes (2)

Proposition

Consider the MS-INARCH(K , L) model and assume that (T-1)–(T-5) hold true.Let (an (·)) be an increasing function of l such that an(l1)− an(l2) −−−→n→∞

∞ for

every l1 > l2 and an(l)/n −−−→n→∞

0 for every l . Then, as n→ ∞, the estimator l

converges in probability to the true number of regimes, i.e.

l P−−−→n→∞

K 0.

The BIC (an (l) = ln(n)l) fulfills the assumptions of Proposition.




Fokianos

Introduction





Consistency



Application

Simulation Study

We simulate data from a MINARCH(2,1) model where the hidden processZt is distributed according to P[Zt = 1] = 1− P[Zt = 0] = 2/3 and foreach regime the Poisson intensities are given by λ1,t = 1.5 + 0.9Yt−1 andλ2,t = 2 + 0.5Yt−1. In this case, the marginal LRT coincides with the LRT.

The goal of this exercise is to compare the AIC and BIC in terms ofdetermining the true number of regimes by counting the percentage oftimes that the aforementioned model selection criteria choose correct thatK = 2 after fitting models with K = 1,2,3,4 regimes. Results are basedon various sample sizes and 1000 runs.

TABLE – Percentage of times that AIC and BIC attain their minimum value among allmodels fitted to a MINACH(2,1) model generated using various sample sizes. Results arebased on 1000 simulations.

Sample Size K = 1 K = 2 K = 3 K = 4AIC BIC AIC BIC AIC BIC AIC BIC

250 61.7 35.5 0 64.1 22.6 0 15.7 0.4500 63.2 19.7 0 80.3 11.3 0 25.5 01000 56.2 3.1 0 96.9 2.5 0 41.3 01500 52.4 0.5 0 99.5 0.5 0 47.1 0




Fokianos

Introduction





Consistency



Application

A Data Example : disease cases caused by E.coli

time

Num

ber o

f Cas

es

2002 2004 2006 2008 2010 2012

040

80

0 5 10 15 20 25

0.0

0.4

0.8

Lag

ACF

0 5 10 15 20 25

−0.1

0.2

0.5

Lag

PACF

FIGURE – Weekly number of reported disease cases caused by E.coli in the state of NorthRhine-Westphalia (Germany) from January 2001 to May 2013 and their sampleautocorrelation and partial autocorrelation functions.




Fokianos

Introduction





Consistency



Application

The data

The time series of E.coli cases consist of 646 observations.We fit several mixture models by assuming that the hidden process (Zt )is :

1 An iid sequence.2 A Markov chain3 Its transition probabilities depends on past values of Yt ’s.

we employ the transformation pk = exp(θk )/(1 + ∑Kj=1 exp(θj )),

k = 1, . . . ,K and θK = 0 to avoid numerical instability.

Lag Simple Poisson Model 1 Model 2 Model 3L K = 1 K = 21 4636.327 4364.537 4361.451 4363.5682 4540.943 4319.091 4380.950 4351.3803 4522.201 4328.674 4357.856 4351.560

TABLE – BIC values after fitting models to E.coli count time series.




Fokianos

Introduction





Consistency



Application

Estimated models

The BIC favors a mixture of a two-component simple mixture model wherein each region Yt is regressed on Yt−1,Yt−2.

The parameter estimates and their standard errors forψ = (θ1,ψ1,0,ψ1,1,ψ1,2,ψ2,0,ψ2,1,ψ2,2)

T are(−0.597,9.475,0.573,0.262,5.431,0.344,0.226)T and(0.068,1.123,0.003,0.003,0.641,0.001,0.002)T .

These results are interpreted as follows. The probability of one regime isp1 = 0.36 and the probability of the other regime is p2 = 1− p1 = 0.64.

In the region which has the smaller probability, the mean number ofweakly cases is larger than the mean number of weakly cases in theregion with higher probability.




Fokianos

Introduction





Consistency



Application

Conclusion

We introduced Markov switching non linear autoregressive Poissonmodels and study their properties.

When the number of regimes is unknown, we showed that a marginallikelihood ratio test for testing the number of hidden regimes converges toa Gaussian process. Note that the computation of such function is muchmore easier when the observations are discrete. The generalization toreal observations is not straightforward.

The MS-INARCH(K , L) model can be extended by introducing a feedbackprocess in the right hand side : MS-INGARCH(K , L1, L2) model.

In this case, the log-likelihood function requires evaluation of all thepossible states for Z1, · · · ,Zn and therefore the complexity of suchcomputation is of the order K n.

An alternative way to deal with this issue is to employ the infiniterepresentation of the MS-INGARCH model in terms of an MS-INARCHmodel and truncate it at some finite order.

However, We can provide a counterexample which shows that estimationof the true number of regimes, under a misspecified GARCH type counttime series model, is not possible.




Fokianos

Introduction





Consistency



Application

Bibliography

Inference in Hidden Markov Models.,O. Cappé, E. Moulines, T. Rydén, Springer, 2005.Asymptotic properties of the maximum likelihood estimator inautoregressive models with Markov regime.,R. Douc, E. Moulines, T. Rydén, Annals of statistics, 32, 2254-2304, 2004.nference and testing for structural change in general Poissonautoregressive models.,P. Doukhan, W. Kengne, Electronic Journal of Statistics, 9, 1267–1314,2015.Mixtures of Nonlinear Poisson Autoregressions,P. Doukhan, K. Fokianos, J. Rynkiewicz, Journal of time series analysis,(to appear).Likelihood ratio inequalities with applications to various mixture,E. Gassiat, Annals of Institute Henri Poincaré 38, 897-906, 2002.Maximum–likelihood estimation for hidden Markov models.,B.G. Leroux, Stochastic Processes and its Applications, 40, 127-143.Asymptotic properties of autoregressive regime-switching models.,M. Olteanu, J. Rynkiewicz, ESAIM P.S., 16, 25-47, 2012.mixture integer-valued ARCH model.,F.Q., Zhu, D. Wang, Journal of Statistical Planning and Inference, 140,2025-2036, 2010.


mixtures of nonlinear poisson autoregressions · 2 days ago · mixtures of nonlinear poisson...

Documents