1

Introduction to Introduction to Discrete-Time Markov ChainDiscrete-Time Markov Chain

2

MotivationMotivation

many dependent systems, e.g., inventory across periods

state of a machine

customers unserved in a distribution system

time

excellent

good

fair

bad

3

MotivationMotivation

any nice limiting results for dependent Xn’s?

no such result for general dependent Xn’s

nice results when Xn’s form a discrete-time

Markov Chain

1 ???

N

nNn

X

N

{ }11

???n

N

X sNn

N

4

Discrete-Time, Discrete-State Discrete-Time, Discrete-State Stochastic ProcessStochastic Process

a stochastic process: a sequence of indexed random variables, e.g., {Xn}, {X(t)}

a discrete-time stochastic process: {Xn}

a discrete-state stochastic process, e.g., state {excellent, good, fair, bad}

set of states {e, g, f, b} {1, 2, 3, 4} {0, 1, 2, 3}

state to describe weather {windy, rainy, cloudy, sunny}

5

Markov PropertyMarkov Property

a discrete-time, discrete-state stochastic process possesses the Markov property if P{Xn+1 = j|Xn = i, Xn−1 = in−1, . . . , X1 = i1, X0 = i0} = pij,

for all i0, i1, …, in1, in, i, j, n 0

time frame: presence n, future n+1, past {i0, i1, …, in1}

meaning of the statement: given presence, the past and the future are conditionally independent

the past and the future are certainly dependent

6

One-Step Transition Probability MatrixOne-Step Transition Probability Matrix

pij 0, i, j 0,

00 01 02

10 11 12

0 1 2

...

...

i i i

p p p

p p p

p p p

P M M M

K

M M M

01, 0,1, 2,...ij

jp i

7

Example 4-1 Example 4-1 Forecasting the WeatherForecasting the Weather

state {rain, not rain}

dynamics of the system rains today rains tomorrow w.p. does not rain today rains tomorrow w.p.

weather of the system across the days, {Xn}

1

1

P

8

Example 4-3 Example 4-3 The Mood of a PersonThe Mood of a Person

mood {cheerful (C), so-so (S), or glum (G)} cheerful today C, S, or G tomorrow w.p. 0.5, 0.4, 0.1 so-so today C, S, or G tomorrow w.p. 0.3, 0.4, 0.3 glum today C, S, or G tomorrow w.p. 0.2, 0.3, 0.5

Xn: mood on the nth day, such that mood {C, S, G}

{Xn}: a 3-state Markov chain (state 0 = C, state 1 = S, state 2 = G) 0.5 0.4 0.1

0.3 0.4 0.3

0.2 0.3 0.5

P

9

Example 4.5Example 4.5A Random Walk ModelA Random Walk Model

a discrete-time Markov chain of number of states {…, -2, -1, 0, 1, 2, …}

random walk: for 0 < p < 1,

pi,i+1 = p = 1 − pi,i−1, i = 0, 1, . . .

10

Example 4.6Example 4.6A Gambling ModelA Gambling Model

each play of a game a gambler gaining $1 w.p. p, and losing $1 o.w.

end of the game: a gambler either broken or accumulating $N transition probabilities:

pi,i+1 = p = 1 − pi,i−1, i = 1, 2, . . . , N − 1; p00 = pNN = 1 example for N = 4

state: Xn, the gambler’s fortune after the n play {0, 1, 2, 3, 4}

1 0 0 0 0

1 0 0 0

0 1 0 0

0 0 1 0

0 0 0 0 1

p p

p p

p p

P

11

Limiting Behavior of Irreducible Chains

12

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

cost of a visit state 1 = $5

state 2 = $8

what is the long-run cost of the above DTMC?

1 2

0.8

0.10.9

0.2

1 { 1} 2 { 2}1

1 1???

n n

N

X XNn

c c

N

13

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

j = fraction of time at state j

N: a very large positive integer

# of periods at state j j N

balance of flow j N i (i N)pij j = i ipij

[ ]ijpP

14

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

j = fraction of time at state j

j = i ipij

1 = 0.91 + 0.22

2 = 0.11 + 0.82

linearly dependent

normalization equation: 1 + 2 = 1

solving: 1 = 2/3, 2 = 1/3

1 2

0.8

0.10.9

C

0.2

( ) 0.666667 0.333333

0.666667 0.333333

P

15

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

1 = 0.752 + 0.013

3 = 0.252

1 + 2 + 3 = 1

1 = 301/801, 2 = 400/801, 3 = 100/801

1 2 3

0.25

0.99

1

0.75

0.01

0 1 0

0.75 0 0.25

0.01 0.99 0

P

0.3757803 0.4993758 0.1248439

0.3757803 0.4993758 0.1248439

0.3757803 0.4993758 0.1248439

P

16

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

an irreducible DTMC {Xn} is positive there exists a unique nonnegative solution to

 

j: stationary (steady-state) distribution of {Xn}

0

0

1 (normalization eqt)

, for all , (balance eqts)

jj

j i iji

p j

17

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

j = fraction of time at state j

j = fraction of expected time at state j

average cost cj for each visit at state j

random i.i.d. Cj for each visit at state j

for aperiodic chain:

1

0lim

k

n

Xk

j jn j

E cc

n

1

0lim ( )

k

n

Xk

j jn j

E CE C

n

0lim ( | )n jn

P X j X

18

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

1 = 301/801, 2 = 400/801, 3 = 100/801

profit per state: c1 = 4, c2 = 8, c3 = -2

average profit

1 2 3

0.25

0.99

1

0.75

0.01

0 1 0

0.75 0 0.25

0.01 0.99 0

P 301 400 100801 801 801

4201801

(4) (8) (2)

19

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

1 = 301/801, 2 = 400/801, 3 = 100/801

C1 ~ unif[0, 8], C2 ~ Geo(1/8), C3 = -4 w.p. 0.5; and = 0 w.p. 0.5 E(C1) = 4, E(C2) = 8, E(C3) = -2

average profit

1 2 3

0.25

0.99

1

0.75

0.01

0 1 0

0.75 0 0.25

0.01 0.99 0

P

301 400 100801 801 801

4201801

(4) (8) (2)