8/12/2019 RP Notes

1/4

I{c} =1 {c}

0 {c}

S

p(n)ij =P r [Xn+1= j | Xn = i] (=pij )

P, P(n)

T()

f(n)ij =P r [Xn = j, Xn1=j, . . . , X 1=j | X0= i]

f(0)ij =

1 i= j

0 i =j

fij =n=1

f(n)ij =P r [ j | X0= i]

f(n)

jj

vjj =n=1

nf(n)jj

Mj =n=1

I{Xn=j}

j i i ji

j

i j Q

T S

y(n)i =P r [X1 T, . . . , X n T | X0= i] =P r

nk=1

{Xk T } | X0= i

, i T

yi = limn

y(n)i

{Xn, n 0} S

i0, i1, i2. . . , in1, i S n 0

P r [Xn+1= j | X0= i0, . . . , X n1= in1, Xn = i] =P r [Xn+1= j | Xn = i]

p(n)ij =p

(1)ij :=pijn 1 i, j S

|S| |S| (i, j) p(n)ij n

P(n)

P

P S

pij i, j Spij >0 i S

jS

pij = 1

8/12/2019 RP Notes

2/4

T()

T

{Xn S, n 0}

n 0

fn : S(n+1)

{0, 1}

I{T()n} = fn(X0(), . . . , X n()) T() n X0(), . . . , X n()

{Xn S, n 0} j S fjj

8/12/2019 RP Notes

3/4

Xn P X0

P r [Xn1 =j1, . . . , X nm =jm] =j0S

p(n1)j0j1

p(n2n1)j1j2

. . . p(nmnm1)jm1jm

fjj =n=0

f(n)jj < 1

E [Mj | X0= j ] =n=1

p(n)jj

Tk, k 1 j

P r [Mj =m | X0= j ] =fjjP r [Mj =m 1 | X0= j ]

P r [XT1+s = k | X0= j, T1< ] =p(s)kj

P r [Mj =m | X0= j ] =fmjj (1 fjj )

E [Mj | X0= j ] = fjj1fjj

fjj 0

E [Mj | X0= j ] =n=1

p(n)jj <

limn

p(n)jj = 0

fjj = 1 P r [Mj =m | X0= j ] = 0

E [Mj | X0= j ] =n=1

p(n)jj =

vjj <

limn

1n

n=1

p(n)jj =j >0

vjj =

limn

1n

n=1

p(n)jj =j = 0

P

y y= Qy,0 y 1

y = 0 y= Qy, 0 y 1

8/12/2019 RP Notes

4/4

y: S R+ A S

/ A,s.ty(j)< infiA y(i)E [y(Xm+1) y(Xm) | Xm = i] 0i / A

fij =fji = 1i, j C y= 0 y= Qy,0 y 1 y :S R+ y(j) j A S

E [y(Xm+1) y(Xm) | Xm = i] 0i / A

s.t= P i = i y: S R+ > 0 A S

E [y(Xm+1) y(Xm) | Xm = i] i / A

E [y(Xm+1) | Xm= i]< i A

B >0, N >0

E [Xk+1 Xk | Xk =i]< i 0

E [Xk+1 Xk | Xk = i]> 0i N

E

(Xk+1 Xk)

+ | Xk = i

Bi N

B >0, N >0

E [Xk+1 Xk | Xk =i]< i 0

E [Xk+1 Xk | Xk = i]> 0i N

m > 0 s.t pij = 0, j = i m, i N