# Stochastic Calculus, Filtering, And Stochastic Control

Post on 06-Apr-2015

214 views

Embed Size (px)

TRANSCRIPT

<p>Stochastic Calculus, Filtering, andStochastic ControlLecture Notes(This version: May 29, 2007)Ramon van HandelSpring 2007PrefaceThese lecture notes were written for the course ACM217: Advanced Topics in Stochas-tic Analysis at Caltech; this year (2007), the topic of this course was stochastic calcu-lus and stochastic control in continuous time. As this is an introductory course on thesubject, and as there are only so many weeks in a term, we will only consider stochas-tic integration with respect to the Wiener process. This is sufcient do develop a largeclass of interesting models, and to develop some stochastic control and ltering theoryin the most basic setting. Stochastic integration with respect to general semimartin-gales, and many other fascinating (and useful) topics, are left for a more advancedcourse. Similarly, the stochastic control portion of these notes concentrates on veri-cation theorems, rather than the more technical existence and uniqueness questions.I hope, however, that the interested reader will be encouraged to probe a little deeperand ultimately to move on to one of several advanced textbooks.I have no illusions about the state of these notesthey were written rather quickly,sometimes at the rate of a chapter a week. I have no doubt that many errors remainin the text; at the very least many of the proofs are extremely compact, and should bemade a little clearer as is betting of a pedagogical (?) treatment. If I have anotheropportunity to teach such a course, I will go over the notes again in detail and attemptthe necessary modications. For the time being, however, the notes are available as-is.If you have any comments at all about these notesquestions, suggestions, omis-sions, general comments, and particularly mistakesI would love to hear from you. Ican be contacted by e-mail at ramon@its.caltech.edu.Required background. I assume that the reader has had a basic course in probabil-ity theory at the level of, say, Grimmett and Stirzaker [GS01] or higher (ACM116/216should be sufcient). Some elementary background in analysis is very helpful.Layout. The LATEX layout was a bit of an experiment, but appears to have beenpositively received. The document is typeset using the memoir package and thedaleif1 chapter style, both of which are freely available on the web.iContentsPreface iContents iiIntroduction 11 Review of Probability Theory 181.1 Probability spaces and events . . . . . . . . . . . . . . . . . . . . . . 191.2 Some elementary properties . . . . . . . . . . . . . . . . . . . . . . 221.3 Random variables and expectation values . . . . . . . . . . . . . . . 241.4 Properties of the expectation and inequalities . . . . . . . . . . . . . 281.5 Limits of random variables . . . . . . . . . . . . . . . . . . . . . . . 311.6 Induced measures, independence, and absolute continuity . . . . . . . 371.7 A technical tool: Dynkins -system lemma . . . . . . . . . . . . . . 431.8 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442 Conditioning, Martingales, and Stochastic Processes 452.1 Conditional expectations and martingales: a trial run . . . . . . . . . 452.2 The Radon-Nikodym theorem revisited . . . . . . . . . . . . . . . . 552.3 Conditional expectations and martingales for real . . . . . . . . . . . 592.4 Some subtleties of continuous time . . . . . . . . . . . . . . . . . . . 652.5 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 693 The Wiener Process 703.1 Basic properties and uniqueness . . . . . . . . . . . . . . . . . . . . 703.2 Existence: a multiscale construction . . . . . . . . . . . . . . . . . . 763.3 White noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 823.4 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 854 The It o Integral 874.1 What is wrong with the Stieltjes integral? . . . . . . . . . . . . . . . 874.2 The It o integral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 924.3 Some elementary properties . . . . . . . . . . . . . . . . . . . . . . 1014.4 The It o calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1024.5 Girsanovs theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 106iiContents iii4.6 The martingale representation theorem . . . . . . . . . . . . . . . . . 1124.7 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1165 Stochastic Differential Equations 1175.1 Stochastic differential equations: existence and uniqueness . . . . . . 1175.2 The Markov property and Kolmogorovs equations . . . . . . . . . . 1215.3 The Wong-Zakai theorem . . . . . . . . . . . . . . . . . . . . . . . . 1255.4 The Euler-Maruyama method . . . . . . . . . . . . . . . . . . . . . . 1305.5 Stochastic stability . . . . . . . . . . . . . . . . . . . . . . . . . . . 1325.6 Is there life beyond the Lipschitz condition? . . . . . . . . . . . . . . 1375.7 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1396 Optimal Control 1416.1 Stochastic control problems and dynamic programming . . . . . . . . 1416.2 Verication: nite time horizon . . . . . . . . . . . . . . . . . . . . . 1486.3 Verication: indenite time horizon . . . . . . . . . . . . . . . . . . 1526.4 Verication: innite time horizon . . . . . . . . . . . . . . . . . . . . 1566.5 The linear regulator . . . . . . . . . . . . . . . . . . . . . . . . . . . 1606.6 Markov chain approximation . . . . . . . . . . . . . . . . . . . . . . 1636.7 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1687 Filtering Theory 1717.1 The Bayes formula . . . . . . . . . . . . . . . . . . . . . . . . . . . 1717.2 Nonlinear ltering for stochastic differential equations . . . . . . . . 1777.3 The Kalman-Bucy lter . . . . . . . . . . . . . . . . . . . . . . . . . 1877.4 The Shiryaev-Wonham lter . . . . . . . . . . . . . . . . . . . . . . 1947.5 The separation principle and LQG control . . . . . . . . . . . . . . . 1977.6 Transmitting a message over a noisy channel . . . . . . . . . . . . . . 2017.7 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2058 Optimal Stopping and Impulse Control 2078.1 Optimal stopping and variational inequalities . . . . . . . . . . . . . 2078.2 Partial observations: the modication problem . . . . . . . . . . . . . 2188.3 Changepoint detection . . . . . . . . . . . . . . . . . . . . . . . . . 2228.4 Hypothesis testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2298.5 Impulse control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2358.6 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240A Problem sets 242A.1 Problem set 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242A.2 Problem set 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244A.3 Problem set 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246A.4 Problem set 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250A.5 Problem set 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252Bibliography 254IntroductionThis course is about stochastic calculus and some of its applications. As the namesuggests, stochastic calculus provides a mathematical foundation for the treatmentof equations that involve noise. The various problems which we will be dealing with,both mathematical and practical, are perhaps best illustrated by considering some sim-ple applications in science and engineering. As we progress through the course, wewill tackle these and other examples using our newly developed tools.Brownian motion, tracking, and nanceBrownian motion and the Wiener processIn 1827, the (then already) famous Scottish botanist Robert Brown observed a rathercurious phenomenon [Bro28]. Brown was interested in the tiny particles found insidegrains of pollen, which he studied by suspending them in water and observing themunder his microscope. Remarkably enough, it appeared that the particles were con-stantly jittering around in the uid. At rst Brown thought that the particles were alive,but he was able to rule out this hypothesis after he observed the same phenomenonwhen using glass powder, and a large number of other inorganic substances, insteadof the pollen particles. A satisfactory explanation of Browns observation was notprovided until the publication of Einsteins famous 1905 paper [Ein05].Einsteins argument relies on the fact that the uid, in which the pollen parti-cles are suspended, consists of a gigantic number of water molecules (though this isnow undisputed, the atomic hypothesis was highly controversial at the time). As theuid is at a nite temperature, kinetic theory suggests that the velocity of every watermolecule is randomly distributed with zero mean value (the latter must be the case, asthe total uid has no net velocity) and is independent from the velocity of the otherwater molecules. If we place a pollen particle in the uid, then in every time inter-val the particle will be bombarded by a large number of water molecules, giving ita net random displacement. The resulting random walk of the particle in the uid isprecisely what Brown observed under his microscope.How should we go about modelling this phenomenon? The following procedure,which is a somewhat modernized version of Einsteins argument, is physically crudebut nonetheless quite effective. Suppose that the pollen particle is bombarded by Nwater molecules per unit time, and that every water molecule contributes an indepen-dent, identically distributed (i.i.d.) random displacement n to the particle (where n1Introduction 20 0.5 1 -1.5-1 -0.500.50 0.5 1-1.5-1-0.500.510 0.5 1-1-0.500.51Figure 0.1. Randomly generated sample paths xt(N) for the Brownian motion model in thetext, with (from left to right) N = 20, 50, 500 collisions per unit time. The displacements nare chosen to be random variables which take the values N1/2with equal probability.has zero mean). Then at time t, the position xt(N) of the pollen particle is given byxt(N) = x0 +|Nt|n=1n.We want to consider the limit where the number of bombardments N is very large,but where every individual water molecule only contributes a tiny displacement to thepollen particlethis is a reasonable assumption, as the pollen particle, while beingsmall, is extremely large compared to a single water molecule. To be concrete, let usdene a constant by var(n) = N1. Note that is precisely the mean-squaredisplacement of the pollen particle per unit time:E(x1(N) x0)2= var</p>
<p> Nn=1n</p>
<p>= N var(n) = .The physical regime in which we are interested now corresponds to the limit N ,i.e., where the number of collisions N is large but the mean-square displacement perunit time remains xed. Writing suggestivelyxt(N) = x0 +t|Nt|n=1 nNt,where n = n</p>
<p>N/ are i.i.d. random variables with zero mean and unit variance,we see that the limiting behavior of xt(N) as N is described by the central limittheorem: we nd that the law of xt(N) converges to a Gaussian distribution with zeromean and variance t. This is indeed the result of Einsteins analysis.The limiting motion of the pollen particle as N is known as Brownian mo-tion. You can get some idea of what xt(N) looks like for increasingly large N byhaving a look at gure 0.1. But now we come to our rst signicant mathematicalproblem: does the limit of the stochastic process t xt(N) as N even existin a suitable sense? This is not at all obvious (we have only shown convergence inIntroduction 3distribution for xed time t), nor is the resolution of this problem entirely straightfor-ward. If we can make no sense of this limit, there would be no mathematical model ofBrownian motion (as we have dened it); and in this case, these lecture notes wouldcome to an end right about here. Fortunately we will be able to make mathematicalsense of Brownian motion (chapter 3), which was rst done in the fundamental workof Norbert Wiener [Wie23]. The limiting stochastic process xt (with = 1) is knownas the Wiener process, and plays a fundamental role in the remainder of these notes.Tracking a diffusing particleUsing only the notion of a Wiener process, we can already formulate one of the sim-plest stochastic control problems. Suppose that we, like Robe...</p>