6.3

COPYRIGHT © 2006 by LAVON B. PAGE

Regular MarkovChains — steady-state probabilitydistributions


Example

T =.8 .2

.4 .6

!

" # #

$

% & &

T2

=.72 .28

.56 .44

!

" # #

$

% & &


Example

T =.8 .2

.4 .6

!

" # #

$

% & &

T2

=.72 .28

.56 .44

!

" # #

$

% & &

T6

=.668 .332

.664 .336

!

" # #

$

% & &

T6 !

2 /3 1/3

2 /3 1/3

"

# $ $

%

& ' '


As n gets larger and larger, Tn gets

closer and closer to the matrix

Example

T =.8 .2

.4 .6

!

" # #

$

% & &

T2

=.72 .28

.56 .44

!

" # #

$

% & &

T6

=.668 .332

.664 .336

!

" # #

$

% & &

T6 !

2 /3 1/3

2 /3 1/3

"

# $ $

%

& ' '

2 /3 1/3

2 /3 1/3

!

" # #

$

% & &


Notice

1 0[ ] 2 /3 1/3

2 /3 1/3

!

" # #

$

% & &

= 2 /3 1/3[ ]


Notice

1 0[ ] 2 /3 1/3

2 /3 1/3

!

" # #

$

% & &

= 2 /3 1/3[ ]

0 1[ ] 2 /3 1/3

2 /3 1/3

!

" # #

$

% & &

= 2 /3 1/3[ ]


Notice

1 0[ ] 2 /3 1/3

2 /3 1/3

!

" # #

$

% & &

= 2 /3 1/3[ ]

0 1[ ] 2 /3 1/3

2 /3 1/3

!

" # #

$

% & &

= 2 /3 1/3[ ]

1 /2 1/2[ ] 2 /3 1/3

2 /3 1/3

!

" # #

$

% & &

= 2 /3 1/3[ ]


no matter what a and b are.

Notice

1 0[ ] 2 /3 1/3

2 /3 1/3

!

" # #

$

% & &

= 2 /3 1/3[ ]

0 1[ ] 2 /3 1/3

2 /3 1/3

!

" # #

$

% & &

= 2 /3 1/3[ ]

1 /2 1/2[ ] 2 /3 1/3

2 /3 1/3

!

" # #

$

% & &

= 2 /3 1/3[ ]

a b[ ] 2 /3 1/3

2 /3 1/3

!

" # #

$

% & &

= 2 /3 1/3[ ]


Moral of the story: No matter whatassumptions you make about the initialprobability distribution, after a largenumber of steps have been taken theprobability distribution is approximately

(2/3 1/3)

Remember T n !

when n is large

2 /3 1/3

2 /3 1/3

!

" # #

$

% & &


Moral of the story: No matter whatassumptions you make about the initialprobability distribution, after a largenumber of steps have been taken theprobability distribution is approximately

(2/3 1/3)

Question: How could we determinethis without computing large powersof T and estimating the limitingmatrix?


Clue: Notice that the probabilitydistribution (2/3 1/3) has the propertythat

2 /3 1/3[ ] .8 .2

.4 .6

!

" # #

$

% & &

= 2 /3 1/3[ ]


is x = 2/3, y = 1/3

In other words, a solution of the matrixequation

Clue: Notice that the probabilitydistribution (2/3 1/3) has the propertythat

2 /3 1/3[ ] .8 .2

.4 .6

!

" # #

$

% & &

= 2 /3 1/3[ ]

x y[ ] .8 .2

.4 .6

!

" # #

$

% & &

= x y[ ]


We can find the steady state probabilitydistribution [ (2/3 1/3) in this example] bysolving for x and y below:

(Remember also that x + y =1)

Idea!!!

x y[ ] .8 .2

.4 .6

!

" # #

$

% & &

= x y[ ]




x y[ ] .8 .2

.4 .6

!

" # #

$

% & &

= x y[ ]

.8x + .4y = x

.2x + .6y = y

x + y = 1




x = 2y

x y[ ] .8 .2

.4 .6

!

" # #

$

% & &

= x y[ ]

.8x + .4y = x

.2x + .6y = y

x + y = 1

.4y = .2x

.2x = .4y




x = 2y

2y + y = 1

x y[ ] .8 .2

.4 .6

!

" # #

$

% & &

= x y[ ]

.8x + .4y = x

.2x + .6y = y

x + y = 1

.4y = .2x

.2x = .4y




x = 2y

2y + y = 1y = 1/3x = 2/3

x y[ ] .8 .2

.4 .6

!

" # #

$

% & &

= x y[ ]

.8x + .4y = x

.2x + .6y = y

x + y = 1

.4y = .2x

.2x = .4y


Example: Bob, Alice and Carol are playingFrisbee. Bob always throws to Alice andAlice always throws to Carol. Carol throwsto Bob 2/3 of the time and to Alice 1/3 ofthe time. In the long run what percentageof the time do each of the players have theFrisbee?


Example: Bob, Alice and Carol are playingFrisbee. Bob always throws to Alice andAlice always throws to Carol. Carol throwsto Bob 2/3 of the time and to Alice 1/3 ofthe time. In the long run what percentageof the time do each of the players have theFrisbee?

B CAA

B

C

T =

0 0 1

1 0 0

1/3 2 /3 0

!

"

# # # #

$

%

& & & &


B CAA

B

C

We must solve the matrix equation

and use the fact that x + y + z = 1

T =

0 0 1

1 0 0

1/3 2 /3 0

!

"

# # # #

$

%

& & & &

x y z[ ]0 0 1

1 0 0

1/3 2 /3 0

!

"

# # # #

$

%

& & & &

= x y z[ ]


We multiply out the left side and equateto the right side:

x y z[ ]0 0 1

1 0 0

1/3 2 /3 0

!

"

# # # #

$

%

& & & &

= x y z[ ]



y + 1/3 z = x

2/3 z = y

x = z

x y z[ ]0 0 1

1 0 0

1/3 2 /3 0

!

"

# # # #

$

%

& & & &

= x y z[ ]



y + 1/3 z = x

2/3 z = y

x = z

Remember also: x + y + z = 1

x y z[ ]0 0 1

1 0 0

1/3 2 /3 0

!

"

# # # #

$

%

& & & &

= x y z[ ]


x + y + z = 1

y + 1/3 z = x

2/3 z = y

x = z

Equations to solve:


x + y + z = 1

y + 1/3 z = x

2/3 z = y

x = z

Solution: x + y + z = 1 can be rewritten as

z + 2/3 z + z = 1

Equations to solve:


x + y + z = 1

y + 1/3 z = x

2/3 z = y

x = z

Equations to solve:

Solution: x + y + z = 1 can be rewritten as

z + 2/3 z + z = 1so z = 3/8

Therefore x = 3/8 and y = 1/4


x = 3/8, y = 1/4, z = 3/8

Alice — 3/8 probability

Bob — 1/4 probability

Carol — 3/8 probability


In the truck rental problem we had thefollowing transition matrix:

NC SC VA

NC

SC

VA

What fraction of the time does a truckspend in each of the 3 states?

.5 .2 .3

.4 .4 .2

.4 .1 .5

!

"

# # # #

$

%

& & & &


remembering that x + y + z = 1.

We have to solve for

[x y z]

.5 .2 .3

.4 .4 .2

.4 .1 .5

!

"

# # # #

$

%

& & & &

= [x y z]


remembering that x + y + z = 1.So the system of equations is

.5x + .4y + .4z = x

.2x + .4y + .1z = y

.3x + .2y + .5z = z

x + y + z =1

We have to solve for

[x y z]

.5 .2 .3

.4 .4 .2

.4 .1 .5

!

"

# # # #

$

%

& & & &

= [x y z]


Rewrite the equations in standard form:

.5x + .4y + .4z = x

.2x + .4y + .1z = y

.3x + .2y + .5z = z

x + y + z =1

x + y + z = 1

!.5x + .4y + .4z = 0

.2x + !.6y + .1z = 0

.3x + .2y + !.5z = 0


Rewrite the equations in standard form:

This system ofequations could besolved using theaugmented matrixmethod of Chapter 1.

.5x + .4y + .4z = x

.2x + .4y + .1z = y

.3x + .2y + .5z = z

x + y + z =1

x + y + z = 1

!.5x + .4y + .4z = 0

.2x + !.6y + .1z = 0

.3x + .2y + !.5z = 0


The methods we have learned in thissection work only with regular Markovchains.


The methods we have learned in thissection work only with regular Markovchains.

A regular Markov Chain is one where forsome positive integer n, the matrix T

n

has no 0 entries.


"

#

$$$$$$$

%

&

'''''''

0 0 1

23

013

12

14

14

"

#

$$$$$$$$$

%

&

'''''''''

16

34

112

13

24

3

16

13

48

4396

2164

43192

T = T 2 =

T 3 =

"

#

$$$$$$$$$

%

&

'''''''''

23

013

16

34

112

724

916

748


T is regularbecause T 3

contains no 0entries.

"

#

$$$$$$$

%

&

'''''''

0 0 1

23

013

12

14

14

"

#

$$$$$$$$$

%

&

'''''''''

16

34

112

13

24

3

16

13

48

4396

2164

43192

T = T 2 =

T 3 =

"

#

$$$$$$$$$

%

&

'''''''''

23

013

16

34

112

724

916

748

6.3

Documents