Download - Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes
![Page 1: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/1.jpg)
Workshop on Stochastic Differential Equations and
Statistical Inference for Markov Processes
January 19th – 22nd 2012Lahore University of Management Sciences
![Page 2: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/2.jpg)
Schedule
• Day 1 (Saturday 21st Jan): Review of Probability and Markov Chains
• Day 2 (Saturday 28th Jan): Theory of Stochastic Differential Equations
• Day 3 (Saturday 4th Feb): Numerical Methods for Stochastic Differential Equations
• Day 4 (Saturday 11th Feb): Statistical Inference for Markovian Processes
![Page 3: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/3.jpg)
Today
• Review of Probability
• Simulation of Random Variables
• Review of Discrete Time Markov Chains
• Review of Continuous Time Markov Chains
![Page 4: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/4.jpg)
REVIEW OF PROBABILITY
![Page 5: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/5.jpg)
Why Probability Models?
• Are laws of nature truly probabilistic?
• Coding uncertainty in models
• Financial Markets, Biological Processes, Turbulence, Statistical Physics, Quantum Physics
![Page 6: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/6.jpg)
Mathematical Foundations– S is a collection of elements (outcomes of an
experiment)– Each (nice) subset of S is an event – A is a collection of (nice) subsets of S– The set function is called a
probability measure iff
![Page 7: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/7.jpg)
Independence
• Two events are independent iff
• This means that the occurrence of one does not affect the occurrence of the other
![Page 8: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/8.jpg)
Conditional Probability• Probability of given that has occurred
• Denoted by
• Independence can be reformulated as =
![Page 9: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/9.jpg)
Random Variables
• A random variable X is areal valued function defined on the sample space
such that
• A is the state space of the random variable
• If A is finite of countably infinite X is discrete• If A is an interval X is continuous
![Page 10: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/10.jpg)
Cumulative Distribution Function
• The cumulative distribution function of X is the function
• F is non decreasing and right continuous and
![Page 11: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/11.jpg)
Probability Mass Function• If X is a discrete random variable, the function
is called the probability mass function of X
• We also have
• The cdf satisfies
![Page 12: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/12.jpg)
Probability Density Function
• If X is a continuous random variable the probability density function is given by
• The cdf satisfies
![Page 13: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/13.jpg)
Discrete Distributions
• Uniform :
• Bernoulli
• Binomial
• Poisson
![Page 14: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/14.jpg)
Continuous Random Variables
• Uniform
• Exponential
• Gaussian
![Page 15: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/15.jpg)
Expectation of a R.V.
• The expectation is defined as
for a continuous random variable
• For a discrete random variable
• What is it?
![Page 16: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/16.jpg)
Expectation of Function of a R.V.
• “Law of the unconscious statistician”
![Page 17: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/17.jpg)
Moments
• The nth moment is given by
• What do they ‘mean’?
![Page 18: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/18.jpg)
Multivariate Distributions• Several random variables can
be associated with the same sample space
• Can define a joint pmf or pdf
• In case of a bivariate random vector•
![Page 19: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/19.jpg)
Marginal pdf
• The marginal pdf of X1 is given by
• The marginal pdf of X2 is given by
![Page 20: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/20.jpg)
Conditional Expectation
• Conditional Expectation is given by
• Note this is a function of a random variable itself!!!
![Page 21: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/21.jpg)
Probability Generating Function
• The pgf of random variable is given by
• The pmf can be recovered by taking derivatives evaluated at 0
![Page 22: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/22.jpg)
Central Limit Theorem
• Why are many physical processes well modeled by Gaussians?
• Let be i.i.d random variables with finite mean and variance then as
the limiting distribution of
is a normal
![Page 23: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/23.jpg)
Law of Large Numbers
• Let be i.i.d random variables with finite mean and variance then
![Page 24: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/24.jpg)
Numerics
• Simulate a 1-D random Walk– Calculate the mean– Calculate the Variance
• Simulate a 2D random walk– Calculate the mean– Calculate the Variance
![Page 25: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/25.jpg)
Simulating a Binomially Distributed Random Variable
• Note sum of Bernoulli trials is a binomial
• Let X i be a Bernoulli trial with probability ‘p’ of success
• is binomial ‘n’, ‘p’
![Page 26: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/26.jpg)
Continuous Random Variables
• Inverse Transform Method– Suppose a random variable has cdf ‘F(x)’– Then Y=F-1(U) also had the same cdf
• Generating the exponential
• Generate the exponential, compare with exact cdf
• Generate a r.v. with cdf
![Page 27: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/27.jpg)
Rejection Method
• Simulate &
• To Simulate look @
• If accept, else reject
• To Simulate N(0,1) let
• If set
![Page 28: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/28.jpg)
Section Challenge
• Kruskal’s Paper and Simulation of the Kruskal Count
• The n-hat problem through various approaches and simulating the n-hat problem
![Page 29: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/29.jpg)
STOCHASTIC PROCESSES
![Page 30: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/30.jpg)
Boring Definitions• A stochastic process is a collection of random
variables– T is the index set, S is the common sample space
• For each fixed denotes a single random variable
•For each fixed is a functions defined on T
![Page 31: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/31.jpg)
Types of Stochastic Processes
• Discrete Time Discrete Space (DTMC)
• Discrete Time Continuous Space (Time Series)
• Continuous Time Discrete Space (CTMC)
• Continuous Time Continuous Space (SDE)
![Page 32: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/32.jpg)
Discrete Time Discrete Space Processes
Discrete Time Markov Chains
![Page 33: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/33.jpg)
Discrete Time Markov Chain
• The index set is discrete (finite or infinite)
• Markov Property
![Page 34: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/34.jpg)
Transition Probability Matrix
• The one step transition probability is defined as
• If the transition probability does not depend on n the process is stationary or homogenous
• The transition matrix is
![Page 35: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/35.jpg)
N-step Transition Probability
• The n step transition probability is
• How is this related to the one step transition probability?
• Guess: Perhaps as the nth power?
![Page 36: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/36.jpg)
Chapman Kolmogorov Equations
• To get from i to j in n steps is equivalent to get from i to k in s steps and from k to j in n-s steps, summed over all possible intermediate k’s
• The n step transitions are just powers of the once step transition!!
![Page 37: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/37.jpg)
Communication Classes
• Two states i and j ‘communicate’ ( ) if for some m and n
• is an equivalence relation
• The set of equivalence classes is called a ‘class’ of the DTMC
• If there is only one class in a MC it is irreducible
![Page 38: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/38.jpg)
Class Properties
• Periodicity : The period of state i, ‘d(i)’; is the GCD of all such n for which
• First Return Time
• Transience & Recurrence– Transience
– Recurrence
![Page 39: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/39.jpg)
Mean Return Time
• Let be the random variable defining the first return time
• The mean of is the mean return time
Transient State Recurrent State
![Page 40: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/40.jpg)
First Passage Time
• First passage time is defined as
![Page 41: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/41.jpg)
Stationary Distribution
• For a DTMC a stationary distribution is non-negative vector
• i.e. the eigenvector of P corresponding to eigenvalue 1
![Page 42: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/42.jpg)
Existence Theorem for Stationary Distribution
• For a positive recurrent, aperiodic and irreducible DTMC there exists a unique stationary distribution such that
![Page 43: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/43.jpg)
Logistic Growth
• The transition probabilities are given by
where
• Note the correspondence with the deterministic model for
![Page 44: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/44.jpg)
DTMC SIS Epidemic Model• Compartmental Model
![Page 45: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/45.jpg)
The Infected Class• I is a random variable that describes the infected class I={0,1,2………N}
• Two classes {0} and {1,2,….N}
• {0} is the absorbing class
• Average time in infected state– F is the sub matrix corresponding to transient states
![Page 46: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/46.jpg)
DTMC SIR Epidemic Model
• The transition probability is given by
with
![Page 47: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/47.jpg)
Section Challenge
• Simulate– Logistic Growth– SIS Model– SIR Model
• Compare mean of MC Simulation with
solution of corresponding deterministic Model
![Page 48: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/48.jpg)
Continuous Time Discrete Space Processes
Continuous Time Markov Chains
![Page 49: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/49.jpg)
Definitions
• The index set is an interval• States are discrete• Markov Property
for any sequence
![Page 50: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/50.jpg)
Transition Probability
• The transition probability is given by
• If this only depends on the length of the time interval chain is homogenous
![Page 51: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/51.jpg)
Chapman Kolmogorov Equations
• The transition probabilities are solutions of the Chapman-Kolmogorov Equations
![Page 52: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/52.jpg)
Waiting Times
• The process stays at state X(0) for a random time W1 then jumps to X(W1)
• Stays in X(W1) for a random time then jumps to X(W2) & so on…..
• The random variable is the waiting time
• Inter-event time
![Page 53: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/53.jpg)
Poisson Process• CTMC with state space {0,1,2,3…….} &– X(0)=0– For Δt sufficiently small
• Satisfies (Kolmogorov Equations)
• i.e. is the Poisson Distribution
![Page 54: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/54.jpg)
Generator Matrix
• Transition rates qji are define in terms of transition probabilities
• The rate matrix or ‘Generator Matrix’ is
![Page 55: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/55.jpg)
Embedded Chain
• If Yn is the DTMC defined by
is known as the embedded chain
• If T=(tji) is the transition matrix of the embedded chain
![Page 56: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/56.jpg)
Class Properties of Embedded Chain
• Many properties carry over form the embedded DTMC to the CTMC
– States that belong to the same class in the DTMC also belong to the same class in the CTMC
– If a state is recurrent in the DTMC so it is in the CTMC– If a class of the DMC is closed so is the class in the
CTMC– If the DTMC is irreducible so is the CTMC– Note: No concept of periodicity in the CTMC!!
![Page 57: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/57.jpg)
Kolmogorov Equations
• The forward equations are given by
• The backward equations are
![Page 58: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/58.jpg)
Stationary Distribution
• For a positive recurrent, irreducible CTMC with generator matrix Q there exists a unique stationary distribution π
such that
Also
![Page 59: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/59.jpg)
Generating Functions and CTMC
• From the Kolmogorov Equations a PDE governing the pgf can be derived
• The RHS consists of P(z,t) and the derivatives of P(z,t)
![Page 60: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/60.jpg)
Interevent Time
• For a CTMC recall the inter-event time was defined as
• The inter-event time has an exponential distribution
![Page 61: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/61.jpg)
SIS Epidemic Model in Continuous Time
• Transition probabilities are given by
![Page 62: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/62.jpg)
SIS CTMC Model
• The Kolmogorov Equations are
where
![Page 63: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/63.jpg)
SIS CTMC Model
• The generator matrix is given by
• The FKE can be written as
![Page 64: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/64.jpg)
SIR Epidemic Model in continuous Time
• The joint probability distribution is given by
![Page 65: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/65.jpg)
Kolmogorov Equations
• The forward equations are given by
where
![Page 66: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/66.jpg)
Asymptotic Results
• For large N and small I(0)=j
![Page 67: Workshop on Stochastic Differential Equations and Statistical Inference for Markov Processes](https://reader035.vdocuments.mx/reader035/viewer/2022062305/56814deb550346895dbb5a8d/html5/thumbnails/67.jpg)
Section Challenge
• Simulate– Logistic Growth– SIS Model– SIR Model
• Compare mean of CTMC with mean of DTMC
and solution of corresponding deterministic Model