markov chains (part 4) - university of...
Post on 30-Nov-2018
229 Views
Preview:
TRANSCRIPT
Markov Chains - 1
Markov Chains (Part 4)
Steady State Probabilities and First Passage Times
Markov Chains - 2
Steady-State Probabilities
• Remember, for the inventory example we had
• For an irreducible ergodic Markov chain, where πj = steady state probability of being in state j
jnijnp !=
"#
)(lim
!!!
"
#
$$$
%
&
=
166.263.285.286.166.263.285.286.166.263.285.286.166.263.285.286.
)8(P
Markov Chains - 3
Some Observations About the Limit
• The behavior of this important limit depends on properties of states i and j and the Markov chain as a whole. – If i and j are recurrent and belong to different classes, then
p(n)ij=0 for all n.
– If j is transient, then for all i. Intuitively, the probability that the Markov chain is in a transient state after a large number of transitions tends to zero.
– In some cases, the limit does not exist! Consider the following Markov chain: if the chain starts out in state 0, it will be back in 0 at times 2,4,6,… and in state 1 at times 1,3,5,…. Thus p(n)
00=1 if n is even and p(n)
00=0 if n is odd. Hence the limit does not exist.
!
limn"#
pij(n ) = 0
!
limn"#
p00(n )
Markov Chains - 4
Steady-State Probabilities • How can we find these probabilities without calculating
P(n) for very large n? • The following are the steady-state equations:
!
" jj= 0
M
# =1
" j = " i piji= 0
M
# for all j = 0,...,M
" j $ 0 for all j = 0,...,M• In matrix notation we have πTP = πT
• Solve a system of linear equations. Note: there are M+2 equations and only M+1 variables (π0, π1, …, πM), so one of the equations is redundant and can be dropped - just don’t drop the equation 1
0=!
=
M
jj"
Solving for the Steady-State Probabilities
• Idea is to go from steady state to steady state:
Markov Chains - 5
!
! TP = ! T and ! i =1i=0
M
!
! 0 !1 ! !M"#
$%
p00 p01 ... p0M
p10 p11 ... p1M
! ! p(M&1)M
pM 0 pM1 ... pMM
"
#
'''''
$
%
(((((
= ! 0 !1 ! !M"#
$%
! 0p00 +!1p10 +!+!M pM 0 = ! 0
! 0p01 +!1p11 +!+!M pM1 = !1
! = !! 0p0M +!1p1M +!+!M pMM = !M
! 0 +!1 +!+!M = 1
Xt
j
i
0
M
t
!
" i
!
"0!
"M
!
" j
Markov Chains - 6
Steady-State Probabilities Examples
• Find the steady-state probabilities for
•
•
• Inventory example
!!!!
"
#
$$$$
%
&
=
43
410
2102
103
231
P
!"#
$%&= 4.06.0
7.03.0P
!!!!
"
#
$$$$
%
&
=
368.0368.0184.0080.00368.0368.0264.000368.0632.0368.0368.0184.0080.0
P
Other Applications of Steady-State Probabilities
• Expected recurrence time: we are often interested in the expected number of steps between consecutive visits to a particular (recurrent) state. – What is the expected number of sunny days between rainy days? – What is the expected number of weeks between ordering cameras?
• Long-run expected average cost per unit time: in many applications, we incur a cost or gain a reward every time a Markov chain visits a specific state. – If we incur costs for carrying inventory, and costs for not meeting
demand, what is the long-run expected average cost per unit time?
Markov Chains - 7
Markov Chains - 8
Expected Recurrence Times
• The expected recurrence time, denoted µjj , is the expected number of transition between two consecutive visits to state j.
• The steady state probabilities, πj , are related to the expected recurrence times, µjj, as
Mjj
jj ,...,1,0 all for 1
=!
=µ
Markov Chains - 9
Weather Example • What is the expected number of sunny days in between
rainy days? • First, calculate πj.
• Now, µ11 = 1/πj = 4 • For this example, we expect 4 sunny days between
rainy days.
0 1
Sun 0Rain 1
0.8 0.20.6 0.4
!
"#
$
%&
! 0 0.8+!10.6 = ! 0! 0 0.2+!10.4 = !1
! 0 +!1 = 1
! 0 =1!!1
1!!1( )0.2+!10.4 = !1
0.2 = 0.8!1!1 =1/ 4 and ! 0 = 3 / 4
Markov Chains - 10
Inventory Example • What is the expected number of weeks in between orders? • First, the steady state probabilities are:
• Now, µ00 = 1/π0 = 3.5 • For this example, on the average, we order cameras every
three and a half weeks.
! 0 = 0.286, !1 = 0.285, ! 2 = 0.263, ! 3 = 0.166
Markov Chains - 11
Expected Recurrence Times Examples
!
P =0.3 0.70.6 0.4"
# $
%
& '
( 0 = 613
(1 = 713
µ00 =13
6 = 2.1667µ11 =13
7 =1.857
!
P =
13
23 0
12 0 1
20 1
43
4
"
#
$ $ $
%
&
' ' '
( 0 = 315
(1 = 415
( 2 = 815
µ00 =15
3 = 5 µ11 =15
4 = 3.75µ22 =15
8 =1.875
0.7 0
0.3 1
0.6
0.4
1
2
2/3 1/2
1/2
3/4 0
1/4
1/3
Markov Chains - 12
Steady-State Cost Analysis
• Once we know the steady-state probabilities, we can do some long-run analyses
• Assume we have a finite-state, irreducible Markov chain • Let C(Xt) be a cost at time t, that is, C(j) = expected cost of being in
state j, for j=0,1,…,M • The expected average cost over the first n time steps is
• The long-run expected average cost per unit time is a function of steady state probabilities
!
E 1n
C Xt( )t =1
n"
#
$ % %
&
' ( (
!
limn"#
E 1n
C Xt( )t =1
n$
%
& ' '
(
) * *
= + jC j( )j=0
M
$
Markov Chains - 13
Steady-State Cost Analysis Inventory Example
• Suppose there is a storage cost for having cameras on hand:
• The long-run expected average cost per unit time is
!
C i( ) =
0 if i = 02 if i =18 if i = 2
18 if i = 3
"
# $ $
% $ $
!
"0C 0( ) +"
1C 1( ) +"
2C 2( ) +"
3C 3( )
= 0.286 0( ) + 0.285 2( ) + 0.268 8( ) + 0.166 18( )= 5.662
Markov Chains - 14
First Passage Times - Motivation • In many applications, we are interested in the time at
which the Markov chain visits a particular state for the first time. – If I start out with a dollar, what is the probability that I will go
broke (for the first time) after n gambles? – If I start out with three cameras in my inventory, what is the
expected number of days after which I will have none for the first time?
• The answers to these questions are related to an important concept called “first passage times”
Markov Chains - 15
First Passage Times • The first passage time from state i to state j is the
number of transitions made by the process in going from state i to state j for the first time
• When i = j, this first passage time is called the recurrence time for state i
• Let fij(n) = probability that the first passage time from state i to state j is equal to n
• What is the difference between fij(n) and pij(n)?
pij(n) includes paths that visit j
fij(n) does not include paths that visit j
Xt
j
i
0 t+n t
Markov Chains - 16
Some Observations about First Passage Times
• First passage times are random variables and have probability distributions associated with them
• fij(n) = probability that the first passage time from state i to state j is equal to n
• These probability distributions can be computed using a simple idea: condition on where the Markov chain goes after the first transition
• For the first passage time from i to j to be n>1, the Markov chain has to transition from i to k (different from j) in one step, and then the first passage time from k to j must be n-1.
• This concept can be used to derive recursive equations for fij(n)
Markov Chains - 17
First Passage Times
The first passage time probabilities satisfy a recursive relationship
Xt
j
i
0
M
t
!
piM
t+1 t+n !
pij
!
pii
!
pi0
!
fMj(n "1)
!
fij(n "1)
!
f0 j(n "1)
fij(1) = pij
(1) = pijfij
(2) = pikk! j" fkj
(1)
!fij
(n) = pikk! j" fkj
(n#1)
Markov Chains - 18
First Passage Times Inventory Example
• Suppose we were interested in the number of weeks until the first order (start in State 3, X0=3)
• Then we would need to know what is the probability that the first order is submitted in – Week 1? – Week 2?
– Week 3?
f301( ) = p30 = 0.080
!
f303( ) = p3k fk0
2( )
k"0# = p31 f10
2( ) + p32 f202( ) + p33 f30
2( )
f30(2) = p3k fk0
1( )
k!0" = p31 f10
(1) + p32 f20(1) + p33 f30
(1)
= p31p10 + p32p20 + p33p30= 0.184(0.632)+ 0.368(0.264)+ 0.368(0.080)= 0.243
Markov Chains - 19
Probability of Ever Reaching j from i
• If the chain starts out in state i, what is the probability that it visits state j at some future time?
• This probability is denoted • If fij=1, then the chain starting at i definitely reaches j at
some future time, in which case f(n)ij is a genuine
probability distribution for the first passage time.
• On the other hand, if fij<1, the chain starting at i may never reach j. In fact, the probability that this happens is 1-fij
fij and fij = fij(n)
n=1
!
"
Markov Chains - 20
Expected First Passage Times
• The expected first passage time from state i to state j is • If fij=1, we can also calculate µij using the idea to
condition on where the chain goes after one transition
!"=
µ+=µM
jkk
kjikij p0
1
µij =! if fij <1
µij = E fij(n)"# $%= nfij
(n) if fij=1 n=1
!
&
µij =1pij + pik (1+µkj )k! j" = pik
k" + pikµkj
k! j" =1+ pikµkj
k! j"
Markov Chains - 21
Expected First Passage Times Inventory Example
• Find the expected time until the first order is submitted
• Find the expected time between orders
!
µ30
=1+ p31µ
10+ p
32µ
20+ p
33µ
30
µ20
=1+ p21µ
10+ p
22µ
20+ p
23µ
30
µ10
=1+ p11µ
10+ p
12µ
20+ p
13µ
30
"
# $
% $
Solve simultaneously,
!
µ10
=1.58 weeksµ
20= 2.51 weeks
µ30
= 3.50 weeks
!
µ00
=1"
0
= 3.50 weeks
top related