lecture notes 7 random processes - stanford universityisl.stanford.edu/~abbas/ee178/lect07-2.pdf ·...

Lecture Notes 7

Random Processes

• Definition

• IID Processes

• Bernoulli Process

◦ Binomial Counting Process

◦ Interarrival Time Process

• Markov Processes

• Markov Chains

◦ Classification of States

◦ Steady State Probabilities

Corresponding pages from B&T: 271–281, 313–340.

EE 178/278A: Random Processes Page 7 – 1

Random Processes

• A random process (also called stochastic process) {X(t) : t ∈ T } is an infinitecollection of random variables, one for each value of time t ∈ T (or, in somecases distance)

• Random processes are used to model random experiments that evolve in time:

◦ Received sequence/waveform at the output of a communication channel

◦ Packet arrival times at a node in a communication network

◦ Thermal noise in a resistor

◦ Scores of an NBA team in consecutive games

◦ Daily price of a stock

◦ Winnings or losses of a gambler

◦ Earth movement around a fault line


Questions Involving Random Processes

• Dependencies of the random variables of the process:

◦ How do future received values depend on past received values?

◦ How do future prices of a stock depend on its past values?

◦ How well do past earth movements predict an earthquake?

• Long term averages:

◦ What is the proportion of time a queue is empty?

◦ What is the average noise power generated by a resistor?

• Extreme or boundary events:

◦ What is the probability that a link in a communication network is congested?

◦ What is the probability that the maximum power in a power distribution lineis exceeded?

◦ What is the probability that a gambler will lose all his capital?


Discrete vs. Continuous-Time Processes

• The random process {X(t) : t ∈ T } is said to be discrete-time if the index setT is countably infinite, e.g., {1, 2, . . .} or {. . . ,−2,−1, 0,+1,+2, . . .}:◦ The process is simply an infinite sequence of r.v.s X1,X2, . . .

◦ An outcome of the process is simply a sequence of numbers

• The random process {X(t) : t ∈ T } is said to be continuous-time if the indexset T is a continuous set, e.g., (0,∞) or (−∞,∞)

◦ The outcomes are random waveforms or random occurances in continuoustime

• We only discuss discrete-time random processes:

◦ IID processes

◦ Bernoulli process and associated processes

◦ Markov processes

◦ Markov chains


IID Processes

• A process X1, X2, . . . is said to be independent and identically distributed (IID,or i.i.d.) if it consists of an infinite sequence of independent and identicallydistributed random variables

• Two important examples:

◦ Bernoulli process: X1,X2, . . . are i.i.d. Bern(p), 0 < p < 1, r.v.s. Model forrandom phenomena with binary outcomes, such as:∗ Sequence of coin flips∗ Noise sequence in a binary symmetric channel∗ The occurrence of random events such as packets (1 corresponding to anevent and 0 to a non-event) in discrete-time

∗ Binary expansion of a random number between 0 and 1

◦ Discrete-time white Gaussian noise (WGN) process: X1,X2, . . . are i.i.d.N (0, N) r.v.s. Model for:∗ Receiver noise in a communication system∗ Fluctuations in a stock price


• Useful properties of an IID process:

◦ Independence: Since the r.v.s in an IID process are independent, any twoevents defined on sets of random variables with non-overlapping indices areindependent

◦ Memorylessness: The independence property implies that the IID process ismemoryless in the sense that for any time n, the future Xn+1,Xn+2, . . . isindependent of the past X1,X2, . . . ,Xn

◦ Fresh start: Starting from any time n, the random process Xn, Xn+1, . . .behaves identically to the process X1, X2, . . ., i.e., it is also an IID processwith the same distribution. This property follows from the fact that the r.v.sare identically distributed (in addition to being independent)


The Bernoulli Process

• The Bernoulli process is an infinite sequence X1,X2, . . . of i.i.d. Bern(p) r.v.s

• The outcome from a Bernoulli process is an infinite sequence of 0s and 1s

• A Bernoulli process is often used to model occurrences of random events;Xn = 1 if an event occurs at time n, and 0, otherwise

• Three associated random processes of interest:

◦ Binomial counting process: The number of events in the interval [1, n]

◦ Arrival time process: The time of event arrivals

◦ Interarrival time process: The time between consecutive event arrivals

• We discuss these processes and their relationships


Binomial Counting Process

• Consider a Bernoulli process X1, X2, . . . with parameter p

• We are often interested in the number of events occurring in some time interval

• For the time interval [1, n], i.e., i = 1, 2, . . . , n, we know that the number ofoccurrences

Wn =

(

n∑

i=1

Xi

)

∼ B(n, p)

• The sequence of r.v.s W1,W2, . . . is referred to as a Binomial counting process

• The Bernoulli process can be obtained from the Binomial counting process as:

Xn = Wn −Wn−1, for n = 1, 2, . . . ,

where W0 = 0

• Outcomes of a Binomial process are integer valued stair-case functions


1 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 192

1

2

34

5

6

7

8

9

10

Wn

Xn

n

0000000000 1 11111111

• Note that the Binomial counting process is not IID

• By the fresh-start property of the Bernoulli process, for any n ≥ 1 and k ≥ 1,the distribution of the number of events in the interval [k + 1, n+ k] is identicalto that of [1, n], i.e., Wn and (Wk+n −Wk) are identically distributed


• Example: Packet arrivals at a node in a communication network can be modeledby a Bernoulli process with p = 0.09.

1. What is the probability that 3 packets arrive in the interval [1, 20], 6 packetsarrive in [1, 40] and 12 packets arrive in [1, 80]?

2. The input queue at the node has a capacity of 103 packets. A packet isdropped if the queue is full. What is the probability that one or more packetsare dropped in a time interval of length n = 104?

Solution: Let Wn be the number of packets arriving in interval [1, n].

1. We want to find the following probability

P{W20 = 3,W40 = 6,W80 = 12},

which is equal to

P{W20 = 3,W40 −W20 = 3,W80 −W40 = 6}

By the independence property of the Bernoulli process this is equal to

P{W20 = 3}P{W40 −W20 = 3}P{W80 −W40 = 6}


Now, by the fresh start property of the Bernoulli process

P{W40 −W20 = 3} = P{W20 = 3}, and

P{W80 −W40 = 6} = P{W40 = 6}

Thus

P{W20 = 3,W40 = 6,W80 = 12} = (P{W20 = 3})2 × P{W40 = 6}

Now, using the Poisson approximation of Binomial, we have

P{W20 = 3} =

(

20

3

)

(0.09)3(0.91)17 ≈ (1.8)3

3!e−1.8 = 0.1607

P{W40 = 6} =

(

40

6

)

(0.09)6(0.91)34 ≈ (3.6)6

6!e−3.6 = 0.0826

Thus

P{W20 = 3,W40 = 6,W80 = 12} ≈ (0.1607)2 × 0.0826 = 0.0021


2. The probability that one or more packets are dropped in a time interval oflength n = 104 is

P{W104 > 103} =10

4

∑

n=1001

(

104

n

)

(0.09)n(0.91)104−n

Difficult to compute, but we can use the CLT!

Since W104 =∑10

4

i=1Xi and E(X) = 0.09 and σ2

X = 0.09× 0.91 = 0.0819,we have

P

104

∑

i=1

Xi > 103

= P

1

100

104

∑

i=1

(Xi − 0.09)√0.0819

>103 − 900

100√0.0819

= P

1

100

104

∑

i=1

(Xi − 0.09)

0.286> 3.5

≈ Q(3.5) = 2× 10−4


Arrival and Interarrival Time Processes

• Again consider a Bernoulli process X1,X2, . . . as a model for random arrivals ofevents

• Let Yk be the time index of the kth arrival, or the kth arrival time, i.e., smallestn such that Wn = k

• Define the interarrival time process associated with the Bernoulli process as

T1 = Y1 and Tk = Yk − Yk−1, for k = 2, 3, . . .

Thus the kth arrival time is given by: Yk = T1 + T2 + . . .+ Tk

000000000000 11111

T1 (Y1) T2 T3 T4 T5

Y2

Y3

n

EE 178/278A: Random Processes – 13

• Let’s find the pmf of Tk:

First, the pmf of T1 is the same as the number of coin flips until a head (i.e, a1) appears. We know that this is Geom(p). Thus T1 ∼ Geom(p)

Now, having an event at time T1, the future is a fresh starting Bernoulli process.Thus, the number of trials T2 until the next event has the same pmf as T1

Moreover, T1 and T2 are independent, since the trials from 1 to T1 areindependent of the trials from T1 + 1 onward. Since T2 is determined exclusivelyby what happens in these future trials, it’s independent of T1

Continuing similarly, we conclude that T1, T2, . . . are i.i.d., i.e., the interarrivalprocess is an IID Geom(p) process

• The interarrival process gives us an alternate definition of a Bernoulli process:

Start with an IID Geom(p) process T1, T2, . . .. Record the arrival of an event attime T1, T1 + T2, T1 + T2 + T3,. . .


• Arrival time process: The sequence of r.v.s Y1, Y2, . . . is denoted by the arrivaltime process. From its relationship to the interarrival time process Y1 = T1,Yk =

∑k

i=1Ti, we can easily find the mean and variance of Yk for any k

E(Yk) = E

(

k∑

i=1

Ti

)

=k∑

i=1

E(Ti) = k × 1

p

Var(Yk) = Var

(

k∑

i=1

Ti

)

=

k∑

i=1

Var(Ti) = k × 1− p

p2

Note that, Y1, Y2, . . . is not an IID process

It is also not difficult to show that the pmf of Yk is

pYk(n) =

(

n− 1

k − 1

)

pk(1− p)n−k for n = k, k + 1, k + 2, . . . ,

which is called the Pascal pmf of order k


• Example: In each minute of a basketball game, Alicia commits a foulindependently with probability p and no foul with probability 1− p. She stopsplaying if she commits her sixth foul or plays a total of 30 minutes. What is thepmf of of Alicia’s playing time?

Solution: We model the foul events as a Bernoulli process with parameter p

Let Z be the time Alicia plays. Then

Z = min{Y6, 30}

The pmf of Y6 is

pY6(n) =

(

n− 1

5

)

p6(1− p)n−6, n = 6, 7, . . .

Thus the pmf of Z is

pZ(z) =

(

z−1

5

)

p6(1− p)z−6, for z = 6, 7, . . . , 29

1−∑29

z=6pZ(z), for z = 30

0, otherwise


Markov Processes

• A discrete-time random process X0, X1, X2, . . ., where the Xns arediscrete-valued r.v.s, is said to be a Markov process if for all n ≥ 0 and all(x0, x1, x2, . . . , xn, xn+1)

P{Xn+1 = xn+1|Xn = xn, . . . ,X0 = x0} = P{Xn+1 = xn+1|Xn = xn},

i.e., the past, Xn−1, . . . , X0, and the future, Xn+1, are conditionallyindependent given the present Xn

• A similar definition for continuous-valued Markov processes can be provided interms of pdfs

• Examples:

◦ Any IID process is Markov

◦ The Binomial counting process is Markov


Markov Chains

• A discrete-time Markov process X0, X1, X2, . . . is called a Markov chain if

◦ For all n ≥ 0, Xn ∈ S , where S is a finite set called the state space.We often assume that S ∈ {1, 2, . . . ,m}

◦ For n ≥ 0 and i, j ∈ S

P{Xn+1 = j|Xn = i} = pij, independent of n

So, a Markov chain is specified by a transition probability matrix

P =

p11 p12 · · · p1mp21 p22 · · · p2m... ... ... ...

pm1 pm2 · · · pmm

Clearly∑m

j=1pij = 1, for all i, i.e., the sum of any row is 1


• By the Markov property, for all n ≥ 0 and all states

P{Xn+1 = j|Xn = i,Xn−1 = in−1, . . . , X0 = i0} = P{Xn+1 = j|Xn = i} = pij

• Markov chains arise in many real world applications:

◦ Computer networks

◦ Computer system reliability

◦ Machine learning

◦ Pattern recognition

◦ Physics

◦ Biology

◦ Economics

◦ Linguistics


Examples

• Any IID process with discrete and finite-valued r.v.s is a Markov chain

• Binary Symmetric Markov Chain: Consider a sequence of coin flips, where eachflip has probability of 1− p of having the same outcome as the previous coinflip, regardless of all previous flips

The probability transition matrix (head is 1 and tail is 0) is

P =

[

1− p pp 1− p

]

A Markov chain can are be specified by a transition probability graph

0 1

p

p

1− p 1− p

Nodes are states, arcs are state transitions; (i, j) from state i to state j (onlydraw transitions with pij > 0)


Can construct this process from an IID process: Let Z1, Z2, . . . be a Bernoulliprocess with parameter p

The Binary symmetric Markov chain Xn can be defined as

Xn+1 = Xn + Zn mod 2, for n = 1, 2, . . . ,

So, each transition corresponds to passing the r.v. Xn through a binarysymmetric channel with additive noise Zn ∼ Bern(p)

• Example: Asymmetric binary Markov chain

State 0 = Machine is working, State 1 = Machine is broken down

The probability transition matrix is

P =

[

0.8 0.20.6 0.4

]

and the transition probability graph is:

0 1

0.2

0.6

0.8 0.4


• Example: Two spiders and a fly

A fly’s possible positions are represented by four states

States 2, 3 : safely flying in the left or right half of a roomState 1: A spider’s web on the left wallState 4: A spider’s web on the right wall

The probability transition matrix is:

P =

1.0 0 0 00.3 0.4 0.3 00 0.3 0.4 0.30 0 0 1.0

The transition probability graph is:

11

0.3

0.30.3

0.4 0.4

0.3

1 2 3 4


• Given a Markov chain model, we can compute the probability of a sequence ofstates given an initial state X0 = i0 using the chain rule as:

P{X1 = i1, X2 = i2, . . . ,Xn = in|X0 = i0} = pi0i1pi1i2 . . . pin−1in

Example: For the spider and fly example,

P{X1 = 2,X2 = 2, X3 = 3|X0 = 2} = p22p22p23

= 0.4× 0.4× 0.3

= 0.048

• There are many other questions of interest, including:

◦ n-state transition probabilities: Beginning from some state i what is theprobability that in n steps we end up in state j

◦ Steady state probabilities: What is the expected fraction of time spent instate i as n → ∞?


n-State Transition Probabilities

• Consider an m-state Markov Chain. Define the n-step transition probabilities as

rij(n) = P{Xn = j|X0 = i} for 1 ≤ i, j ≤ m

• The n-step transition probabilities can be computed using theChapman-Kolmogorov recursive equation:

rij(n) =

m∑

k=1

rik(n− 1)pkj for n > 1, and all 1 ≤ i, j ≤ m,

starting with rij(1) = pij

This can be readily verified using the law of total probability


• We can view the rij(n), 1 ≤ i, j ≤ m, as the elements of a matrix R(n), calledthe n-step transition probability matrix, then we can view theKolmogorov-Chapman equations as a sequence of matrix multiplications:

R(1) = PR(2) = R(1)P = PP = P2

R(3) = R(2)P = P3

...

R(n) = R(n− 1)P = Pn


• Example: For the binary asymmetric Markov chain

R(1) = P =

[

0.8 0.20.6 0.4

]

R(2) = P2 =

[

0.76 0.240.72 0.28

]

R(3) = P3 =

[

0.752 0.2480.744 0.256

]

R(4) = P4 =

[

0.7504 0.24960.7488 0.2512

]

R(5) = P5 =

[

0.7501 0.24990.7498 0.2502

]

In this example, each rij seems to converge to a non-zero limit independent ofthe initial state, i.e., each state has a steady state probability of being occupiedas n → ∞


• Example: Consider the spiders-and-fly example

P =

1.0 0 0 00.3 0.4 0.3 00 0.3 0.4 0.30 0 0 1.0

P2 =

1.0 0 0 00.42 0.25 0.24 0.090.09 0.24 0.25 0.420 0 0 1.0

P3 =

1.0 0 0 00.50 0.17 0.17 0.160.16 0.17 0.17 0.500 0 0 1.0

P4 =

1.0 0 0 00.55 0.12 0.12 0.210.21 0.12 0.12 0.550 0 0 1.0


As n → ∞, we obtain

P∞ =

1.0 0 0 02/3 0 0 1/31/3 0 0 2/30 0 0 1.0

Here rij converges, but the limit depends on the initial state and can be 0 forsome states

Note that states 1 and 4 corresponding to capturing the fly by one of the spidersare absorbing states, i.e., they are infinitely repeated once visitedThe probability of being in non-absorbing states 2 and 3 diminishes as timeincreases


Classification of States

• As we have seen, various states of a Markov chain can have differentcharacteristics

• We wish to classify the states by the long-term frequency with which they arevisited

• Let A(i) be the set of states that are accessible from state i (may include iitself), i.e., can be reached from i in n steps, for some n

• State i is said to be recurrent if starting from i, any accessible state j must besuch that i is accessible from j , i.e., j ∈ A(i) iff i ∈ A(j). Clearly, this impliesthat if i is recurrent then it must be in A(i)

• A state is said to be transient if it is not recurrent

• Note that recurrence/transcience is determined by the arcs (transitions withnonzero probability), not by actual values of probabilities


• Example: Classify the states of the following Markov chain

1 2 3 4 5

• The set of accessible states A(i) from some recurrent state i is called arecurrent class

• Every state k in a recurrent class A(i) is recurrent and A(k) = A(i)

Proof: Suppose i is recurrent and k ∈ A(i). Then k is accessible from i andhence, since i is recurrent, k can access i and hence, through i, any state inA(i). Thus A(i) ⊆ A(k)

Since i can access k and hence any state in A(k), A(k) ⊆ A(i). ThusA(i) = A(k). This argument also proves that any k ∈ A(i) is recurrent


• Two recurrent classes are either identical or disjoint

• Summary:

◦ A Markov chain can be decomposed into one or more recurrent classes pluspossibly some transient states

◦ A recurrent state is accessible from all states in its class, but it is notaccessible from states in other recurrent classes

◦ A transient state is not accessible from any recurrent state

◦ At least one recurrent state must be accessible from a given transient state


• Example: Find the recurrent classes in the following Markov chains

1 2

2

2

3

3

3

4

4

41


Periodic Classes

• A recurrent class A is called periodic if its states can be grouped into d > 1disjoint subsets S1, S2, . . . , Sd, ∪d

i=1Si = A, such that all transitions from onesubset lead to the next subset

6

1

2

3

4

5

S1

S2

S3


• Example: Consider a Markov chain with probability transition graph

2

3

1

Note that the recurrent class {1, 2} is periodic and for i = 1, 2

rii(n) =

{

1 if n is even0 otherwise

• Note that if the class is periodic, rii(n) never converges to a steady-state


Steady State Probabilities

• Steady-state convergence theorem: If a Markov chain has only one recurrentclass and it is not periodic, then rij(n) tends to a steady-state πj independent

of i, i.e.,lim

n→∞rij(n) = πj for all i

• Steady-state equations: Taking the limit as n → ∞ of theChapman-Kolmogorov equations

rij(n+ 1) =m∑

k=1

rik(n)pkj for 1 ≤ i, j ≤ m,

we obtain the set of linear equations, called the balance equations:

πj =m∑

k=1

πkpkj for j = 1, 2, . . . ,m

The balance equations together with the normalization equation

m∑

j=1

πj = 1,


uniquely determine the steady state probabilities π1, π2, . . . , πm

• The balance equations can be expressed in a matrix form as:

ΠP = Π, where Π = [π1 π2 . . . , πm]

• In general there are m− 1 linearly independent balance equations

• The steady state probabilities form a probability distribution over the statespace, called the stationary distribution of the chain. If we set P{X0 = j} = πj

for j = 1, 2, . . . ,m, we have P{Xn = j} = πj for all n ≥ 1 and j = 1, 2, . . . ,m

• Example: Binary Symmetric Markov Chain

P =

[

1− p pp 1− p

]

Find the steady-state probabilities

Solution: We need to solve the balance and normalization equations

π1 = p11π1 + p21π2 = (1− p)π1 + pπ2

π2 = p12π1 + p22π2 = pπ1 + (1− p)π2

1 = π1 + π2


Note that the first two equations are linearly dependent. Both yield π1 = π2.Substituting in the last equation, we obtain π1 = π2 = 1/2

• Example: Asymmetric binary Markov chain

P =

[

0.8 0.20.6 0.4

]

The steady state equations are:

π1 = p11π1 + p21π2 = 0.8π1 + 0.6π2, and 1 = π1 + π2

Solving the two equations yields π1 = 3/4 and π2 = 1/4

• Example: Consider the Markov chain defined by the following probabilitytransition graph

1 2 3 40.90.1

0.5

0.5

0.80.2

0.6

0.4


• Note: The solution of the steady state equations yields πj = 0 if a state istransient, and πj > 0 if a state is recurrent


Long Term Frequency Interpretations

• Let vij(n) be the # of times state j is visited beginning from state i in n steps

• For a Markov chain with a single aperiodic recurrent class, can show that

limn→∞

vij(n)

n= πj

Since this result doesn’t depend on the starting state i, πj can be interpreted asthe long-term frequency of visiting state j

• Since each time state j is visited, there is a probability pjk that the nexttransition is to state k, πjpjk can be interpreted as the long-term frequency oftransitions from j to k

• These frequency interpretations allow for a simple interpretation of the balanceequations, that is, the long-term frequency πj is the sum of the long-termfrequencies πkpkj of transitions that lead to j

πj =m∑

k=1

πkpkj


• Another interpretation of the balance equations: Rewrite the LHS of the balanceequation as

πj = πj

m∑

k=1

pjk

=m∑

k=1

πjpjk = πjpjj +m∑

k=1,k 6=j

πjpjk

The RHS can be written as

m∑

k=1

πkpkj = πjpjj +m∑

k=1,k 6=j

πkpkj

Subtracting the pjjπj from both sides yields

m∑

k=1,k 6=j

πkpkj =m∑

k=1,k 6=j

πjpjk, j = 1, 2, . . . ,m


The long-term frequency of transitions into j is equal to the long-termfrequency of transitions out of j

j

1 1

2 2

mm

π1p1j

π2p2j

πmpmj

πjpj1

πjpj2

πjpjm

This interpretation is similar to Kirkoff’s current law. In general, if we partitiona chain (with single aperiodic recurrent class) into two sets of states, thelong-term frequency of transitions from the first set to the second is equal to thelong-term frequency of transitions from the second to the first


Birth-Death Processes

• A birth-death process is a Markov chain in which the states are linearly arrangedand transitions can only occur to a neighboring state, or else leave the stateunchanged

• For a birth-death Markov chain use the notation:

bi = P{Xn+1 = i+ 1|Xn = i}, birth probability at state i

di = P{Xn+1 = i− 1|Xn = i}, death probability at state i

0 1 m− 1 m

1− b0

b0

1− b1 − d1

b1

d1 d2 dm−1 dm

bm−2 bm−1

1− bm−1 − dm−1 1− dm


• For a birth-death process the balance equations can be greatly simplified: Cutthe chain between states i− 1 and i. The long-term frequency of transitionsfrom right to left must be equal to the long-term frequency of transitions fromleft to right, thus:

πidi = πi−1bi−1, or πi = πi−1

bi−1

di

• By recursive substitution, we obtain

πi = π0

b0b1 . . . bi−1

d1d2 . . . di, i = 1, 2, . . . ,m

To obtain the steady state probabilities we use these equations together with thenormalization equation

1 =m∑

j=0

πj


Examples

• Queuing: Packets arrive at a communication node with buffer size m packets.Time is discretized in small periods. At each period:

◦ If the buffer has less than m packets, the probability of 1 packet added to itis b, and if it has m packets, the probability of adding another packet is 0

◦ If there is at least 1 packet in the buffer, the probability of 1 packet leaving itis d > b, and if it has 0 packets, this probability is 0

◦ If the number of packets in the buffer is from 1 to m− 1, the probability ofno change in the state of the buffer is 1− b− d. If the buffer has no packets,the probability of no change in the state is 1− b, and if there are m packetsin the buffer, this probability is 1− d

We wish to find the long-term frequency of having i packets in the queue


We introduce a birth-death Markov chain with states 0, 1, . . . ,m, correspondingto the number of packets in the buffer

0 1 m− 1 m

1− b

b

1− b− d

b

d d d d

b b

1− b− d 1− d

The local balance equations are

πid = πi−1b, i = 1, . . . ,m

Define ρ = b/d < 1, then πi = ρπi−1, which leads to

πi = ρiπ0

Using the normalizing equation:∑m

i=0πi = 1, we obtain

π0(1 + ρ+ ρ2 + · · ·+ ρm) = 1


Hence for i = 1, . . . ,m

πi =ρi

1 + ρ+ ρ2 + · · ·+ ρm

Using the geometric progression formula, we obtain

πi = ρi1− ρ

1− ρm+1

Since ρ < 1, πi → ρi(1− ρ) as m → ∞, i.e., {πi} converges to Geometric pmf

• The Ehrenfest model: This is a Markov chain arising in statistical physics. Itmodels the diffusion through a membrane between two containers. Assume thatthe two containers have a total of 2a molecules. At each step a molecule isselected at random and moved to the other container (so a molecule diffuses atrandom through the membrane). Let Yn be the number of molecules incontainer 1 at time n and Xn = Yn − a. Then Xn is a birth-death Markovchain with 2a+ 1 states; i = −a,−a+ 1, . . . ,−1, 0, 1, 2, . . . , a and probabilitytransitions

pij =

bi = (a− i)/2a, if j = i+ 1di = (a+ i)/2a, if j = i− 10, otherwise


The steady state probabilities are given by:

πi = π−a

b−ab−a+1 . . . bi−1

d−a+1d−a+2 . . . di, i = −a,−a+ 1, . . . ,−1, 0, 1, 2, . . . , a

= π−a

2a(2a− 1) . . . (a− i+ 1)

1× 2× . . .× (a+ i)

= π−a

2a!

(a+ i)!(2a− (a+ i))!=

(

2a

a+ i

)

π−a

Now the normalization equation gives

a∑

i=−a

(

2a

a+ i

)

π−a = 1

Thus, π−a = 2−2a

Substituting, we obtain

πi =

(

2a

a+ i

)

2−2a


lecture notes 7 random processes - stanford universityisl.stanford.edu/~abbas/ee178/lect07-2.pdf ·...

Documents