cs145: probability & computingcs.brown.edu/courses/csci1450/lectures/lec19_mabsorb.pdfcs145:...

CS145: Probability & ComputingLecture 19: Markov Chains & Absorption,

Instructor: Eli UpfalBrown University Computer Science

Figure credits:Bertsekas & Tsitsiklis, Introduction to Probability, 2008

Pitman, Probability, 1999

CS145: Lecture 19 OutlineØMarkov Chain Classes and Equilibrium DistributionsØAbsorption in Markov ChainsØNext Subject: Randomized Algorithms and

Probabilistic Analysis

Finite Markov ChainsX0 X1 X2 X3 · · ·

Sec. 6.1 Discrete-Time Markov Chains 3

We will generally allow the probabilities pii to be positive, in which case it ispossible for the next state to be the same as the current one. Even though thestate does not change, we still view this as a state transition of a special type (a“self-transition”).

Specification of Markov Models

• A Markov chain model is specified by identifying(a) the set of states S = {1, . . . , m},(b) the set of possible transitions, namely, those pairs (i, j) for which

pij > 0, and,(c) the numerical values of those pij that are positive.

• The Markov chain specified by this model is a sequence of randomvariables X0, X1, X2, . . ., that take values in S and which satisfy

P(Xn+1 = j |Xn = i, Xn−1 = in−1, . . . , X0 = i0) = pij ,

for all times n, all states i, j ∈ S, and all possible sequences i0, . . . , in−1of earlier states.

All of the elements of a Markov chain model can be encoded in a transitionprobability matrix, which is simply a two-dimensional array whose elementat the ith row and jth column is pij :

⎡

⎢⎢⎣

p11 p12 · · · p1mp21 p22 · · · p2m...

......

...pm1 pm2 · · · pmm

⎤

⎥⎥⎦ .

It is also helpful to lay out the model in the so-called transition probabilitygraph, whose nodes are the states and whose arcs are the possible transitions.By recording the numerical values of pij near the corresponding arcs, one canvisualize the entire model in a way that can make some of its major propertiesreadily apparent.

Example 6.1. Alice is taking a probability class and in each week she can beeither up-to-date or she may have fallen behind. If she is up-to-date in a givenweek, the probability that she will be up-to-date (or behind) in the next week is0.8 (or 0.2, respectively). If she is behind in the given week, the probability thatshe will be up-to-date (or behind) in the next week is 0.6 (or 0.4, respectively). Weassume that these probabilities do not depend on whether she was up-to-date orbehind in previous weeks, so the problem has the typical Markov chain character(the future depends on the past only through the present).

pij = P (Xt+1 = j | Xt = i)

Xt 2 {1, . . . ,m}Markov Property: Given the current state, the past & future are independent.Finite State:

P =

State Transition Matrix: State Transition Diagram:

Example Generic convergence questions:

• Does rij(n) converge to something?0 .5 0 .8

0 .5 0 .5 0 .5

1 2 3

21 1 1

0.2n odd: r2 2(n)= n even: r2 2(n)=

n = 0 n = 1 n = 2 n = 100 n = 101 Does the limit depend on initialr11(n)

• state?

r12(n) 0 .4

r21(n) 3 41 20 .3 0 .3

r22(n)r (n)=11

r (n)=31

r (n)=21

Recurrent and transient states

• State i is recurrent if:starting from i,and from wherever you can go,there is a way of returning to i

• If not recurrent, called transient

3 4

6 75

1 2

8

– i transient:P(Xn = i)⇥ 0,i visited finite number of times

• Recurrent class:collection of recurrent states that“communicate” with each otherand with no other state

2

P (X0, X1, . . . , Xn) = P (X0)nY

t=1

P (Xt | Xt�1)

Multi-Step State Transitions⇡ti = P (Xt = i) ⇡t = [⇡t1,⇡t2, . . . ,⇡tm]

T

After n time steps:


We will generally allow the probabilities pii to be positive, in which case it ispossible for the next state to be the same as the current one. Even though thestate does not change, we still view this as a state transition of a special type (a“self-transition”).

Specification of Markov Models

• A Markov chain model is specified by identifying(a) the set of states S = {1, . . . , m},(b) the set of possible transitions, namely, those pairs (i, j) for which

pij > 0, and,(c) the numerical values of those pij that are positive.

• The Markov chain specified by this model is a sequence of randomvariables X0, X1, X2, . . ., that take values in S and which satisfy

P(Xn+1 = j |Xn = i, Xn−1 = in−1, . . . , X0 = i0) = pij ,

for all times n, all states i, j ∈ S, and all possible sequences i0, . . . , in−1of earlier states.

All of the elements of a Markov chain model can be encoded in a transitionprobability matrix, which is simply a two-dimensional array whose elementat the ith row and jth column is pij :

⎡

⎢⎢⎣

p11 p12 · · · p1mp21 p22 · · · p2m...

......

...pm1 pm2 · · · pmm

⎤

⎥⎥⎦ .

It is also helpful to lay out the model in the so-called transition probabilitygraph, whose nodes are the states and whose arcs are the possible transitions.By recording the numerical values of pij near the corresponding arcs, one canvisualize the entire model in a way that can make some of its major propertiesreadily apparent.

Example 6.1. Alice is taking a probability class and in each week she can beeither up-to-date or she may have fallen behind. If she is up-to-date in a givenweek, the probability that she will be up-to-date (or behind) in the next week is0.8 (or 0.2, respectively). If she is behind in the given week, the probability thatshe will be up-to-date (or behind) in the next week is 0.6 (or 0.4, respectively). Weassume that these probabilities do not depend on whether she was up-to-date orbehind in previous weeks, so the problem has the typical Markov chain character(the future depends on the past only through the present).

pij = P (Xt+1 = j | Xt = i)

P =

State Transition Matrix: State Transition Diagram:



0 .5 0 .5 0 .5

1 2 3

21 1 1

0.2n odd: r2 2(n)= n even: r2 2(n)=


• state?

r12(n) 0 .4

r21(n) 3 41 20 .3 0 .3

r22(n)r (n)=11

r (n)=31

r (n)=21




3 4

6 75

1 2

8



2

⇡Tn = ⇡Tn�1P = ⇡

Tn�2PP = ⇡

T0 P

n

Steady-State (Equilibrium) Distribution⇡ti = P (Xt = i) ⇡t = [⇡t1,⇡t2, . . . ,⇡tm]

T

After n time steps:⇡Tn = ⇡

Tn�1P = ⇡

Tn�2PP = ⇡

T0 P

n

If state distribution converges:

The equilibrium distribution is the unique solution to m+1 linear equations:

Alternative Interpretation: Eigenvector of state transition matrix whose eigenvalue is 1.

⇡⇤j =mX

i=1

⇡⇤i pij , j = 1, . . . ,m.

||⇡n � ⇡n�1|| ! 0

mX

i=1

⇡⇤i = 1.

⇡⇤ = PT⇡⇤

Conditions for Steady-State DistributionA unique steady-state distribution exists if and only if:Ø The Markov chain is aperiodic (not periodic).Ø The Markov chain is irreducible (single recurrent class).

⇡⇤j =mX

i=1

⇡⇤i pij , j = 1, . . . ,m.

mX

i=1

⇡⇤i = 1.

Properties of steady-state distributions:Ø For any transient state i,Ø For all state pairs (i,j),

⇡⇤i = 0limn!1

rij(n) = ⇡⇤j

Linear equations defining unique steady-state distribution:

12 Markov Chains Chap. 6

6

1

2

3

4

5

S1

S3

S2

Figure 6.8: Structure of a periodic recurrent class.

Note that given a periodic recurrent class, a positive time n, and a state j inthe class, there must exist some state i such that rij(n) = 0. The reason is that,from the definition of periodicity, the states are grouped in subsets S1, . . . , Sd,and the subset to which j belongs can be reached at time n from the states inonly one of the subsets. Thus, a way to verify aperiodicity of a given recurrentclass R, is to check whether there is a special time n ≥ 1 and a special states ∈ R that can be reached at time n from all initial states in R, i.e., ris(n) > 0for all i ∈ R. As an example, consider the first chain in Fig. 6.7. State s = 2can be reached at time n = 2 starting from every state, so the unique recurrentclass of that chain is aperiodic.

A converse statement, which we do not prove, also turns out to be true:if a recurrent class is not periodic, then a time n and a special state s with theabove properties can always be found.

Periodicity

Consider a recurrent class R.

• The class is called periodic if its states can be grouped in d > 1disjoint subsets S1, . . . , Sd, so that all transitions from Sk lead to Sk+1(or to S1 if k = d).

• The class is aperiodic (not periodic) if and only if there exists a timen and a state s in the class, such that pis(n) > 0 for all i ∈ R.

Violations of Steady-State ConditionsPeriodic (not aperiodic) Markov chains:

Multiple recurrent classes (not irreducible):

Sec. 6.2 Classification of States 11

1 2 3 4

Single class of recurrent states

1 2

3

Single class of recurrent states (1 and 2)and one transient state (3)

Two classes of recurrent states (class of state1 and class of states 4 and 5)and two transient states (2 and 3)

1 2 3 4 5

Figure 6.7: Examples of Markov chain decompositions into recurrent classes andtransient states.

to the presence or absence of a certain periodic pattern in the times that a stateis visited. In particular, a recurrent class is said to be periodic if its states canbe grouped in d > 1 disjoint subsets S1, . . . , Sd so that all transitions from onesubset lead to the next subset; see Fig. 6.8. More precisely,

if i ∈ Sk and pij > 0, then{

j ∈ Sk+1, if k = 1, . . . , d − 1,j ∈ S1, if k = d.

A recurrent class that is not periodic, is said to be aperiodic.Thus, in a periodic recurrent class, we move through the sequence of subsets

in order, and after d steps, we end up in the same subset. As an example, therecurrent class in the second chain of Fig. 6.7 (states 1 and 2) is periodic, andthe same is true of the class consisting of states 4 and 5 in the third chain of Fig.6.7. All other classes in the chains of this figure are aperiodic.



0 .5 0 .5 0 .5

1 2 3

21 1 1

0.2n odd: r2 2(n)= n even: r2 2(n)=


• state?

r12(n) 0 .4

r21(n) 3 41 20 .3 0 .3

r22(n)r (n)=11

r (n)=31

r (n)=21




3 4

6 75

1 2

8



2

1 2 3

Both aperiodic and irreducible:Sec. 6.2 Classification of States 11

1 2 3 4


1 2

3



1 2 3 4 5




j ∈ Sk+1, if k = 1, . . . , d − 1,j ∈ S1, if k = d.



Sec. 6.2 Classification of States 11

1 2 3 4


1 2

3



1 2 3 4 5




j ∈ Sk+1, if k = 1, . . . , d − 1,j ∈ S1, if k = d.



Transient state:⇡⇤3 = 0

ExampleGenericconvergencequestions:

•Doesrij(n)convergetosomething?0.50.8

0.50.50.5

123

2 111

0.2n odd:r22(n)=n even:r22(n)=

n=0n=1n=2n=100n=101Doesthelimitdependoninitialr11(n)

•state?

r12(n)0.4

r21(n)34 120.30.3

r22(n)r(n)= 11

r(n)= 31

r(n)= 21

Recurrentandtransientstates

•Stateiisrecurrentif:startingfromi,andfromwhereveryoucango,thereisawayofreturningtoi

•Ifnotrecurrent,calledtransient

34

675

12

8

–itransient:P(Xn=i)⇥0,ivisitedfinitenumberoftimes

•Recurrentclass:collectionofrecurrentstatesthat“communicate”witheachotherandwithnootherstate

2

ExampleGenericconvergencequestions:

•Doesrij(n)convergetosomething?0.50.8

0.50.50.5

123

2 111

0.2n odd:r22(n)=n even:r22(n)=

n=0n=1n=2n=100n=101Doesthelimitdependoninitialr11(n)

•state?

r12(n)0.4

r21(n)34 120.30.3

r22(n)r(n)= 11

r(n)= 31

r(n)= 21

Recurrentandtransientstates

•Stateiisrecurrentif:startingfromi,andfromwhereveryoucango,thereisawayofreturningtoi

•Ifnotrecurrent,calledtransient

34

675

12

8

–itransient:P(Xn=i)⇥0,ivisitedfinitenumberoftimes

•Recurrentclass:collectionofrecurrentstatesthat“communicate”witheachotherandwithnootherstate

2

Example: Card ShufflingCard Shu✏ing• Markov chain states: all n! ordered of n distinct cards.• Transitions: Pick a random card and move it to top of thedeck.

• LemmaThe uniform distribution is a stationary distribution of this chain.

Proof.

Consider a state C = c1, . . . , cn. The chain can move to this statefrom the n states C (j) = c2, . . . , c1, . . . , cn, where card c1 is inplace j of the deck. There are n such states, and each hasprobability 1/n to move to state C .

⇡C =nX

j=1

⇡C(j)1

n

1

n!=

nX

j=1

1

n

1

n!

Ø Markov chain states: All n! orderings of n distinct cardsØ Transitions: Pick one of the n cards uniformly at random,

and move that card to the top of the deck.Ø Stationary distribution: Uniform distribution over permutations

How many moves till the cards are (almost) mixed? Inadvance class

Example: Birth-Death Processes

Steady-State Probabilities Visit frequency interpretation

• Do the rij(n) converge to some �j?(independent of the initial state i)

�j = �kpkj• Yes, if:

�

k

– recurrent states are all in a single class,and • (Long run) frequency of being in j: �j

– single recurrent class is not periodic• Frequency of transitions k ⇤ j: �kpkj

• Assuming “yes,” start from key recursionFrequency of transitions into j:

rij(n) = rik(n� 1)pkj•

k

�� kpkj

k

1 πjpjj – take the limit as n⇤⌅ π1p1j

�j =�

�kpkj, for all jk 2

π2p2j j– Additional equation:

. . .

. . .

��j = 1

j πmpmj m

Example Birth-death processes

1- p - q1- p 1 1 1- q0 m

0 .5 0 .8p p0 1

0 1 2 3 .. . m0 .5

q1 q q2 m

21 pi

i+10.2 i �ipi = �i+1qi+1

qi+1

• Special case: pi = p and qi = q for all i⇥ = p/q =load factor

p�i+1 = �i = �i⇥q

�i =i�0⇥ , i = 0,1, . . . , m

• Assume p < q and m ⇥ ⌅

�0 = 1� ⇥⇥

E[Xn] = (in steady-state)1� ⇥


be viewed as the long-term expected fraction of transitions that move the statefrom j to k.†

Expected Frequency of a Particular Transition

Consider n transitions of a Markov chain with a single class which is aperi-odic, starting from a given initial state. Let qjk(n) be the expected numberof such transitions that take the state from j to k. Then, regardless of theinitial state, we have

limn�⇥

qjk(n)n

= �jpjk.

Given the frequency interpretation of �j and �kpkj , the balance equation

�j =m�

k=1

�kpkj

has an intuitive meaning. It expresses the fact that the expected frequency �jof visits to j is equal to the sum of the expected frequencies �kpkj of transitionsthat lead to j; see Fig. 7.13.

Figure 7.13: Interpretation of the balance equations in terms of frequencies. Ina very large number of transitions, we expect a fraction �kpkj that bring the statefrom k to j. (This also applies to transitions from j to itself, which occur withfrequency �jpjj .) The sum of the expected frequencies of such transitions is theexpected frequency �j of being at state j.

† In fact, some stronger statements are also true, such as the following. Wheneverwe carry out a probabilistic experiment and generate a trajectory of the Markov chainover an infinite time horizon, the observed long-term frequency with which state j isvisited will be exactly equal to �j , and the observed long-term frequency of transitionsfrom j to k will be exactly equal to �jpjk. Even though the trajectory is random, theseequalities hold with essential certainty, that is, with probability 1.

2




�

k





k

�� kpkj

k


�j =�



. . .

. . .

��j = 1

j πmpmj m


1- p - q1- p 1 1 1- q0 m

0 .5 0 .8p p0 1

0 1 2 3 .. . m0 .5

q1 q q2 m

21 pi

i+10.2 i �ipi = �i+1qi+1

qi+1


p�i+1 = �i = �i⇥q

�i =i�0⇥ , i = 0,1, . . . , m


�0 = 1� ⇥⇥






limn�⇥

qjk(n)n

= �jpjk.


�j =m�

k=1

�kpkj




2




�

k





k

�� kpkj

k


�j =�



. . .

. . .

��j = 1

j πmpmj m


1- p - q1- p 1 1 1- q0 m

0 .5 0 .8p p0 1

0 1 2 3 .. . m0 .5

q1 q q2 m

21 pi

i+10.2 i �ipi = �i+1qi+1

qi+1


p�i+1 = �i = �i⇥q

�i =i�0⇥ , i = 0,1, . . . , m


�0 = 1� ⇥⇥






limn�⇥

qjk(n)n

= �jpjk.


�j =m�

k=1

�kpkj




2

Equilibrium state distributionis a geometric distribution!

Theorem: In a stationary distribution, for any partition (C,D) of the state set the probability of moving from C to D must be equal to the probability of moving from D to C

The PageRank Citation Ranking: Bringing Order to the Web

WWW Conference - November 1998

Lawrence Page, Sergey Brin, Rajeev Motwan, Terry Winograd

Abstract

The importance of a Web page is an inherently subjective matter, which

depends on the readers interests, knowledge and attitudes. But there is

still much that can be said objectively about the relative importance of

Web pages. This paper describes PageRank, a mathod for rating Web

pages objectively and mechanically, effectively measuring the human

interest and attention devoted to them. We compare PageRank to an

idealized random Web surfer. We show how to efficiently compute

PageRank for large numbers of pages. And, we show how to apply

PageRank to search and to user navigation.

Example: Google’s PagerankConsider the following Markov chain:Ø One state for each of m webpagesØ At each time step,

choose one of the outgoing links from a page with equal probability

Ø Rank of a page: Fraction of time that a “random surfer” spends on that page (equilibrium distribution)

Wikipedia

Good Properties:Ø Webpages are important if many

other pages link to themØ Webpages are important if other

important pages link to them

Example: Google’s PagerankConsider the following Markov chain:Ø One state for each of m webpagesØ At each time step,


Ø Rank of a page: Fraction of time that a “random surfer” spends on that page (equilibrium distribution)

Wikipedia

Problems:Ø There may be many absorbing

states, so the desired equilibrium distribution does not exist!

Ø Results can be sensitive to single link changes, and thus “noisy”

Example: Google’s PagerankConsider the following Markov chain:Ø At each time step, with probability q,


Ø At each time step, with probability 1-q,choose one of m pages uniformly

Ø Rank of a page: Fraction of time that a “random surfer” spends on that page

Wikipedia

Good Properties:Ø Webpages are important if many

other pages link to themØ Webpages are important if other

important pages link to themØ Steady-state distribution always exists

Example: Google’s Pagerank

Wikipedia

Because web graph is sparse, it is possible to compute the probabilities of reaching each page even if the number of pages is very large.

⇡Tn = ⇡Tn�1P = ⇡

Tn�2PP = ⇡

T0 P

n

CS145: Lecture 19 OutlineØMarkov Chain Classes and Equilibrium DistributionsØAbsorption in Markov Chains

Transient and Absorbing States

Calculating absorption probabilities Expected time to absorption

• What is the probability ai that:process eventually settles in state 4,

1

given that the initial state is i? 3 0 .5

440 .50 .4

1 54

0.60 .2

0 .2 1 21

3 0 .3 0 .8

440 .50 .4

• Find expected number of transitions µ0.6

i,0 .2 until reaching the absorbing state,

1 2

given that the initial state is i?0 .8

For i = 4, ai =For i = 5, a = µi = 0 for i =i

⇥ For all other i: µ= for all other i

= 1 + pijµjai pijaj, ij

⇥

j

– unique solution– unique solution

Mean first passage and recurrence

times

• Chain with one recurrent class;fix s recurrent

• Mean first passage time from i to s:

ti = E[min{n ⇤ 0 such that Xn = s} |X0 = i]

• t1, t2, . . . , tm are the unique solution to

ts = 0,

ti = 1 +⇥

pij tj, for all i = sj

• Mean recurrence time of s:

t⇥s = E[min{n ⇤ 1 such that Xn = s} |X0 = s]

• t⇥s = 1 +�

j psj tj

⌃

2

Consider a simplified class of Markov chainswhere all states are one of two types:Ø Transient: Visited finite number of times.Ø Absorbing: Once entered, process never

leaves (self-transition probability equals 1).Practical examples of such Markov chains: Ø Games with win and loss states.Ø Computer systems with failure states.

To characterize such Markov chains: Ø From each transient state, what is the probability of each absorbing state?Ø From each transient state, what is the expected time until absorption?


1 20.3

1

0.4

3 4 1

0.4

0.3

0.30.3

0.3

10.4

pij

32

4

1 32 41.0

1.0

000

0 000

00.40.30.30.3

Figure 6.2: The transition probability graph and the transition probability ma-trix in Example 6.2, for the case where m = 4.

To verify this property, note that

P(X0 = i0, X1 = i1, . . . , Xin = in)= P(Xn = in |X0 = i0, . . . , Xn−1 = in−1)P(X0 = i0, . . . , Xn−1 = in−1)= pin−1inP(X0 = i0, . . . , Xn−1 = in−1),

where the last equality made use of the Markov property. We then apply thesame argument to the term P(X0 = i0, . . . , Xn−1 = in−1) and continue similarly,until we eventually obtain the desired expression. If the initial state X0 is givenand is known to be equal to some i0, a similar argument yields

P(X1 = i1, . . . , Xin = in |X0 = i0) = pi0i1pi1i2 · · · pin−1in .

Graphically, a state sequence can be identified with a sequence of arcs in thetransition probability graph, and the probability of such a path (given the ini-tial state) is given by the product of the probabilities associated with the arcstraversed by the path.

Example 6.3. For the spider and fly example (Example 6.2), we have

P(X1 = 2, X2 = 2, X3 = 3, X4 = 4 |X0 = 2) = p22p22p23p34 = (0.4)2(0.3)2.

We also have

P(X0 = 2, X1 = 2, X2 = 2, X3 = 3, X4 = 4) = P(X0 = 2)p22p22p23p34

= P(X0 = 2)(0.4)2(0.3)2.

Note that in order to calculate a probability of this form, in which there is noconditioning on a fixed initial state, we need to specify a probability law for theinitial state X0.

Computing Absorption Probabilities


associated probabilities may depend on the starting state. In the sequel, we fix aparticular absorbing state, denoted by s, and consider the absorption probabilityai that s is eventually reached, starting from i:

ai = P(Xn eventually becomes equal to the absorbing state s |X0 = i).Absorption probabilities can be obtained by solving a system of linear equations,as indicated below.

Absorption Probability Equations

Consider a Markov chain in which each state is either transient or absorbing.We fix a particular absorbing state s. Then, the probabilities ai of eventuallyreaching state s, starting from i, are the unique solution of the equations

as = 1,ai = 0, for all absorbing i ̸= s,

ai =m∑

j=1

pijaj , for all transient i.

The equations as = 1, and ai = 0, for all absorbing i ̸= s, are evidentfrom the definitions. To verify the remaining equations, we argue as follows. Letus consider a transient state i and let A be the event that state s is eventuallyreached. We haveai = P(A |X0 = i)

=m∑

j=1

P(A |X0 = i, X1 = j)P(X1 = j |X0 = i) (total probability thm.)

=m∑

j=1

P(A |X1 = j)pij (Markov property)

=m∑

j=1

ajpij .

The uniqueness property of the solution of the absorption probability equationsrequires a separate argument, which is given in the theoretical problems section.

The next example illustrates how we can use the preceding method tocalculate the probability of entering a given recurrent class (rather than a givenabsorbing state).

Example 6.11. Consider the Markov chain shown in Fig. 6.17(a). We wouldlike to calculate the probability that the state eventually enters the recurrent class







ai =m∑

j=1



=m∑

j=1


=m∑

j=1


=m∑

j=1

ajpij .




Theorem:







ai =m∑

j=1



=m∑

j=1


=m∑

j=1


=m∑

j=1

ajpij .




Proof: A = {state s is eventually reached}


1 20.3

1

0.4

3 4 1

0.4

0.3

0.30.3

0.3

10.4

pij

32

4

1 32 41.0

1.0

000

0 000

00.40.30.30.3








P(X1 = 2, X2 = 2, X3 = 3, X4 = 4 |X0 = 2) = p22p22p23p34 = (0.4)2(0.3)2.

We also have

P(X0 = 2, X1 = 2, X2 = 2, X3 = 3, X4 = 4) = P(X0 = 2)p22p22p23p34

= P(X0 = 2)(0.4)2(0.3)2.


Computing Absorption Probabilities







ai =m∑

j=1



=m∑

j=1


=m∑

j=1


=m∑

j=1

ajpij .










ai =m∑

j=1



=m∑

j=1


=m∑

j=1


=m∑

j=1

ajpij .




Theorem:


1 20.3

1

0.4

3 4 1

0.4

0.3

0.30.3

0.3

10.4

pij

32

4

1 32 41.0

1.0

000

0 000

00.40.30.30.3








P(X1 = 2, X2 = 2, X3 = 3, X4 = 4 |X0 = 2) = p22p22p23p34 = (0.4)2(0.3)2.

We also have

P(X0 = 2, X1 = 2, X2 = 2, X3 = 3, X4 = 4) = P(X0 = 2)p22p22p23p34

= P(X0 = 2)(0.4)2(0.3)2.


From each initial state i, what is the probability ai of eventually reaching state 1?

a1 = 1 a4 = 0a2 =2

3a3 =

1

3

Example: The Gambler’s Ruin


independent. The gambler plays continuously until he either accumulates a tar-get amount of $m, or loses all his money. What is the probability of eventuallyaccumulating the target amount (winning) or of losing his fortune?

We introduce the Markov chain shown in Fig. 6.18 whose state i representsthe gambler’s wealth at the beginning of a round. The states i = 0 and i = mcorrespond to losing and winning, respectively.

All states are transient, except for the winning and losing states which areabsorbing. Thus, the problem amounts to finding the probabilities of absorptionat each one of these two absorbing states. Of course, these absorption probabilitiesdepend on the initial state i.

0 4

p

31 2

1 - p

pp

1 - p 1 - p WinLose

Figure 6.18: Transition probability graph for the gambler’s ruin problem(Example 6.12). Here m = 4.

Let us set s = 0 in which case the absorption probability ai is the probabilityof losing, starting from state i. These probabilities satisfy

a0 = 1,

ai = (1 − p)ai−1 + pai+1, i = 1, . . . , m − 1,

am = 0 .

These equations can be solved in a variety of ways. It turns out there is an elegantmethod that leads to a nice closed form solution.

Let us write the equations for the ai as

(1 − p)(ai−1 − ai) = p(ai − ai+1), i = 1, . . . , m − 1.

Then, by denoting

δi = ai − ai+1, i = 1, . . . , m − 1,

and

ρ =1 − p

p,

the equations are written as

δi = ρδi−1, i = 1, . . . , m − 1,

from which we obtain

δi = ρiδ0, i = 1, . . . , m − 1.

Ø At each round, win $1 with probability p, lose $1 with probability (1-p)Ø Gambler plays until wins a target of $m, or loses all moneyØ Starting with $i, what is the probability that the gambler wins or loses?

m = 4

Sec. 6.4 Absorption Probabilities and Expected Time to Absorption 29

This, together with the equation δ0 + δ1 + · · · + δm−1 = a0 − am = 1, implies that

(1 + ρ + · · · + ρm−1)δ0 = 1.

Thus, we have

δ0 =

⎧⎨

⎩

1 − ρ1 − ρm if ρ ̸= 1,1m

if ρ = 1,

and, more generally,

δi =

⎧⎪⎨

⎪⎩

ρi(1 − ρ)1 − ρm if ρ ̸= 1,1m

if ρ = 1.

From this relation, we can calculate the probabilities ai. If ρ ̸= 1, we have

ai = a0 − δi−1 − · · ·− δ0= 1 − (ρi−1 + · · · + ρ + 1)δ0

= 1 − 1 − ρi

1 − ρ ·1 − ρ

1 − ρm ,

= 1 − 1 − ρi

1 − ρm ,

and finally the probability of losing, starting from a fortune i, is

ai =ρi − ρm

1 − ρm , i = 1, . . . , m − 1.

If ρ = 1, we similarly obtain

ai =m − i

m.

The probability of winning, starting from a fortune i, is the complement 1−ai,and is equal to

1 − ai =

⎧⎪⎨

⎪⎩

1 − ρi

1 − ρm if ρ ̸= 1,im

if ρ = 1.

The solution reveals that if ρ > 1, which corresponds to p < 1/2 and unfa-vorable odds for the gambler, the probability of losing approaches 1 as m → ∞regardless of the size of the initial fortune. This suggests that if you aim for a largeprofit under unfavorable odds, financial ruin is almost certain.


independent. The gambler plays continuously until he either accumulates a tar-get amount of $m, or loses all his money. What is the probability of eventuallyaccumulating the target amount (winning) or of losing his fortune?

We introduce the Markov chain shown in Fig. 6.18 whose state i representsthe gambler’s wealth at the beginning of a round. The states i = 0 and i = mcorrespond to losing and winning, respectively.

All states are transient, except for the winning and losing states which areabsorbing. Thus, the problem amounts to finding the probabilities of absorptionat each one of these two absorbing states. Of course, these absorption probabilitiesdepend on the initial state i.

0 4

p

31 2

1 - p

pp

1 - p 1 - p WinLose

Figure 6.18: Transition probability graph for the gambler’s ruin problem(Example 6.12). Here m = 4.

Let us set s = 0 in which case the absorption probability ai is the probabilityof losing, starting from state i. These probabilities satisfy

a0 = 1,

ai = (1 − p)ai−1 + pai+1, i = 1, . . . , m − 1,

am = 0 .

These equations can be solved in a variety of ways. It turns out there is an elegantmethod that leads to a nice closed form solution.

Let us write the equations for the ai as

(1 − p)(ai−1 − ai) = p(ai − ai+1), i = 1, . . . , m − 1.

Then, by denoting

δi = ai − ai+1, i = 1, . . . , m − 1,

and

ρ =1 − p

p,

the equations are written as

δi = ρδi−1, i = 1, . . . , m − 1,

from which we obtain

δi = ρiδ0, i = 1, . . . , m − 1.

Let ai be probability of loss, 1� ai be probability of win.

Detailed analysis in B&T Example 7.11.

Expected Time to Absorption



We now turn our attention to the expected number of steps until a recurrentstate is entered (an event that we refer to as “absorption”), starting from aparticular transient state. For any state i, we denote

µi = E[number of transitions until absorption, starting from i

]

= E[min{n ≥ 0|Xn is recurrent}

∣∣ X0 = i].

If i is recurrent, this definition sets µi to zero.We can derive equations for the µi by using the total expectation theorem.

We argue that the time to absorption starting from a transient state i is equalto 1 plus the expected time to absorption starting from the next state, whichis j with probability pij . This leads to a system of linear equations which isstated below. It turns out that these equations have a unique solution, but theargument for establishing this fact is beyond our scope.

Equations for the Expected Time to Absorption

The expected times µi to absorption, starting from state i are the uniquesolution of the equations

µi = 0, for all recurrent states i,

µi = 1 +m∑

j=1

pijµj , for all transient states i.

Example 6.13. (Spiders and Fly) Consider the spiders-and-fly model of Ex-ample 6.2. This corresponds to the Markov chain shown in Fig. 6.19. The statescorrespond to possible fly positions, and the absorbing states 1 and m correspondto capture by a spider.

Let us calculate the expected number of steps until the fly is captured. Wehave

µ1 = µm = 0,

and

µi = 1 + 0.3 · µi−1 + 0.4 · µi + 0.3 · µi+1, for i = 2, . . . , m − 1.

We can solve these equations in a variety of ways, such as for example bysuccessive substitution. As an illustration, let m = 4, in which case, the equationsreduce to

µ2 = 1 + 0.4 · µ2 + 0.3 · µ3, µ3 = 1 + 0.3 · µ2 + 0.4 · µ3.

Theorem:



We now turn our attention to the expected number of steps until a recurrentstate is entered (an event that we refer to as “absorption”), starting from aparticular transient state. For any state i, we denote

µi = E[number of transitions until absorption, starting from i

]

= E[min{n ≥ 0|Xn is recurrent}

∣∣ X0 = i].

If i is recurrent, this definition sets µi to zero.We can derive equations for the µi by using the total expectation theorem.

We argue that the time to absorption starting from a transient state i is equalto 1 plus the expected time to absorption starting from the next state, whichis j with probability pij . This leads to a system of linear equations which isstated below. It turns out that these equations have a unique solution, but theargument for establishing this fact is beyond our scope.

Equations for the Expected Time to Absorption

The expected times µi to absorption, starting from state i are the uniquesolution of the equations

µi = 0, for all recurrent states i,

µi = 1 +m∑

j=1

pijµj , for all transient states i.

Example 6.13. (Spiders and Fly) Consider the spiders-and-fly model of Ex-ample 6.2. This corresponds to the Markov chain shown in Fig. 6.19. The statescorrespond to possible fly positions, and the absorbing states 1 and m correspondto capture by a spider.

Let us calculate the expected number of steps until the fly is captured. Wehave

µ1 = µm = 0,

and

µi = 1 + 0.3 · µi−1 + 0.4 · µi + 0.3 · µi+1, for i = 2, . . . , m − 1.

We can solve these equations in a variety of ways, such as for example bysuccessive substitution. As an illustration, let m = 4, in which case, the equationsreduce to

µ2 = 1 + 0.4 · µ2 + 0.3 · µ3, µ3 = 1 + 0.3 · µ2 + 0.4 · µ3.


1 20.3

1

0.4

3 4 1

0.4

0.3

0.30.3

0.3

10.4

pij

32

4

1 32 41.0

1.0

000

0 000

00.40.30.30.3








P(X1 = 2, X2 = 2, X3 = 3, X4 = 4 |X0 = 2) = p22p22p23p34 = (0.4)2(0.3)2.

We also have

P(X0 = 2, X1 = 2, X2 = 2, X3 = 3, X4 = 4) = P(X0 = 2)p22p22p23p34

= P(X0 = 2)(0.4)2(0.3)2.


From each initial state i, what is the expected number of steps to absorption?

µ1 = 0 µ4 = 0µ2 = µ3 =10

3

Matches mean of a geometric distribution with success probability of 0.3.

Absorption in General Markov Chains


{4, 5} starting from one of the transient states. For the purposes of this problem,the possible transitions within the recurrent class {4, 5} are immaterial. We cantherefore lump the states in this recurrent class and treat them as a single absorbingstate (call it state 6); see Fig. 6.17(b). It then suffices to compute the probabilityof eventually entering state 6 in this new chain.

1

1 2 3 4 51

0.30.4

0.20.3

0.5

1

0.3 0.7

1 2 3 6

0.30.4

0.20.8

1

(a)

(b)

0.2

0.1

0.1

0.2

Figure 6.17: (a) Transition probability graph in Example 6.11. (b) A newgraph in which states 4 and 5 have been lumped into the absorbing states = 6.

The absorption probabilities ai of eventually reaching state s = 6 startingfrom state i, satisfy the following equations:

a2 = 0.2a1 + 0.3a2 + 0.4a3 + 0.1a6,

a3 = 0.2a2 + 0.8a6.

Using the facts a1 = 0 and a6 = 1, we obtain

a2 = 0.3a2 + 0.4a3 + 0.1,

a3 = 0.2a2 + 0.8.

This is a system of two equations in the two unknowns a2 and a3, which can bereadily solved to yield a2 = 21/31 and a3 = 29/31.

Example 6.12. (Gambler’s Ruin) A gambler wins $1 at each round, withprobability p, and loses $1, with probability 1 − p. Different rounds are assumed


{4, 5} starting from one of the transient states. For the purposes of this problem,the possible transitions within the recurrent class {4, 5} are immaterial. We cantherefore lump the states in this recurrent class and treat them as a single absorbingstate (call it state 6); see Fig. 6.17(b). It then suffices to compute the probabilityof eventually entering state 6 in this new chain.

1

1 2 3 4 51

0.30.4

0.20.3

0.5

1

0.3 0.7

1 2 3 6

0.30.4

0.20.8

1

(a)

(b)

0.2

0.1

0.1

0.2

Figure 6.17: (a) Transition probability graph in Example 6.11. (b) A newgraph in which states 4 and 5 have been lumped into the absorbing states = 6.

The absorption probabilities ai of eventually reaching state s = 6 startingfrom state i, satisfy the following equations:

a2 = 0.2a1 + 0.3a2 + 0.4a3 + 0.1a6,

a3 = 0.2a2 + 0.8a6.

Using the facts a1 = 0 and a6 = 1, we obtain

a2 = 0.3a2 + 0.4a3 + 0.1,

a3 = 0.2a2 + 0.8.

This is a system of two equations in the two unknowns a2 and a3, which can bereadily solved to yield a2 = 21/31 and a3 = 29/31.

Example 6.12. (Gambler’s Ruin) A gambler wins $1 at each round, withprobability p, and loses $1, with probability 1 − p. Different rounds are assumed

Ø First phase: Find probability of reaching (being absorbed by) each recurrent class

Ø Second phase: Find equilibrium distribution of states within each of these recurrent classes

cs145: probability & computingcs.brown.edu/courses/csci1450/lectures/lec19_mabsorb.pdfcs145:...

Documents