probability: the classical limit theoremsassets.cambridge.org/97811070/53212/frontmatter/... ·...

Probability: The Classical Limit Theorems

The theory of probability has been extraordinarily successful at describing a variety ofnatural phenomena, from the behavior of gases to the transmission of information, andis a powerful tool with applications throughout mathematics. At its heart are a numberof concepts familiar in one guise or another to many: Gauss’ bell-shaped curve, thelaw of averages, and so on, concepts that crop up in so many settings that they are, insome sense, universal. This universality is predicted by probability theory to aremarkable degree. It is the aim of the book to explain the theory, prove classical limittheorems, and investigate their ramifications.

The author assumes a good working knowledge of basic analysis, real and complex.From this, he maps out a route from basic probability, via random walks, Brownianmotion, the law of large numbers and the central limit theorem, to aspects of ergodictheorems, equilibrium and nonequilibrium statistical mechanics, communication overa noisy channel, and random matrices. Numerous examples and exercises enrich thetext.

H E N RY M C K E A N is a professor in the Courant Institute of Mathematical Sciences atNew York University. He is a fellow of the American Mathematical Society and in2007 he received the Leroy P. Steele Prize for his life’s work.

www.cambridge.org© in this web service Cambridge University Press

Cambridge University Press978-1-107-05321-2 - Probability: The Classical Limit TheoremsHenry MckeanFrontmatterMore information

http://www.cambridge.org/9781107053212

http://www.cambridge.org


Probability:The Classical Limit Theorems

HENRY MCKEANNew York University






University Printing House, Cambridge CB2 8BS, United Kingdom

Cambridge University Press is part of the University of Cambridge.

It furthers the University’s mission by disseminating knowledge in the pursuit ofeducation, learning and research at the highest international levels of excellence.

www.cambridge.orgInformation on this title: www.cambridge.org/9781107053212

c© Henry McKean 2014

This publication is in copyright. Subject to statutory exceptionand to the provisions of relevant collective licensing agreements,no reproduction of any part may take place without the written

permission of Cambridge University Press.

First published 2014

Printed in the United Kingdom by CPI Group Ltd, Croydon CR0 4YY

A catalogue record for this publication is available from the British Library

Library of Congress Cataloguing in Publication dataMcKean, Henry P. (Henry Pratt), 1930- author.

Probability : the classical limit theorems / Henry Mckean, New York University.pages cm

1. Limit theorems (Probability theory) I. Title.QA273.67.M35 2014

519.2–dc232014022165

ISBN 978-1-107-05321-2 HardbackISBN 978-1-107-62827-4 Paperback

Cambridge University Press has no responsibility for the persistence or accuracy ofURLs for external or third-party internet websites referred to in this publication,

and does not guarantee that any content on such websites is, or will remain,accurate or appropriate.






v

Il y a des faussetes deguisees qui representent si bien la verite, que ce seraitmal juger de ne s’y laisser pas tromper.

La Rochefoucauld, Maximes no. 282






vi

DEDICATION

To the memory of M. Kac, W. Feller, K. Ito, N. Levinson, and GretchenWarren, who taught me so much about everything, and to my dear wife Rasa.






Contents

Preface page xviiGuide xx

1 Preliminaries 11.1 Lebesgue measure: an outline 1

1.1.1 Caratheodory’s lemma 11.1.2 Measurable sets 21.1.3 The integral 4

1.2 Probabilities and expectations 51.2.1 Reduction to Lebesgue measure 51.2.2 Expectation 61.2.3 Independence 7

1.3 Conditional probabilities and expectations 91.3.1 Signed measures 91.3.2 Radon–Nikodym 101.3.3 Conditional probabilities and expectations 111.3.4 Games, fair and otherwise 15

1.4 Rademacher functions and Wiener’s trick 191.5 Stirling’s approximation 22

1.5.1 The poor man’s Stirling 221.5.2 Kelvin’s method 24

1.6 Fast Fourier: series and integrals 251.6.1 Fourier series 261.6.2 Fourier integral 281.6.3 Poisson summation 291.6.4 Several dimensions 31

1.7 Distribution functions and densities 321.7.1 Compactness 32

vii






viii Contents

1.7.2 Convolution 331.7.3 Fourier transforms 341.7.4 Laplace transform 351.7.5 Some special densities 35

1.8 More probability: some useful tools illustrated 361.8.1 Chebyshev’s Inequality 361.8.2 Mills’s Ratio 371.8.3 Doob’s Inequality 371.8.4 Kolmogorov’s 01 Law 381.8.5 Borel–Cantelli Lemmas 39

2 Bernoulli Trials 422.1 The law of large numbers (LLN) 43

2.1.1 The weak law 432.1.2 The individual or strong law 432.1.3 Application of the weak law: polynomial

approximation 442.1.4 Application of the strong law: empirical

distributions 452.2 The Gaussian approximation (CLT) 482.3 The law of iterated logarithm 512.4 Large deviations 54

2.4.1 Cramer’s estimate 542.4.2 Empirical distributions 552.4.3 The descent 562.4.4 Legendre–Fenchel duality 582.4.5 General variables 58

3 The Standard Random Walk 603.1 The Markov property 61

3.1.1 The simple Markov property 613.1.2 The strict Markov property 62

3.2 Passage times 643.2.1 A better way: the reflection principle of

D. Andre. 663.2.2 Yet another take: stopping 683.2.3 Passage to a distance 683.2.4 Two-sided passage: the Gambler’s Ruin 69

3.3 Loops 713.3.1 Equidistribution 713.3.2 The actual number of visits 73






Contents ix

3.3.3 Long runs 743.4 The arcsine law 76

3.4.1 The conditional arcsine law 813.4.2 Drift and the conventional wisdom 82

3.5 Volume 83

4 The Standard Random Walk in Higher Dimensions 874.1 What RW(2) and RW(3) do as n ↑ ∞ 874.2 How RW(3) escapes to ∞ 91

4.2.1 Speed 914.2.2 Direction 92

4.3 Gauss–Landen, Polya and RW(2) 944.4 RW(2): loops and occupation numbers 97

4.4.1 Duration of a large number of loops 994.4.2 Long runs 101

4.5 RW(2): a hitting distribution 1014.6 RW(2): volume 1044.7 RW(3): hitting probabilities 105

4.7.1 The meaning of G 1064.7.2 Comparison with R3 1074.7.3 Electrostatics 1084.7.4 Back to Z3 1084.7.5 Energy and capacity in R3 1094.7.6 Finale in Z3 1104.7.7 Grounding 1114.7.8 Harmonic functions 111

4.8 RW(3): volume 1124.9 Non-negative harmonic functions 115

4.9.1 Standard walks: first pass 1154.9.2 Standard walks: second pass 1184.9.3 Variants of RW(1) 1204.9.4 Space-time walks, mostly in dimension 1 126

5 LLN, CLT, Iterated Log, and Arcsine in General 1325.1 LLN 132

5.1.1 Another way 1335.1.2 Kolmogorov’s 1933 proof 1335.1.3 Doob’s proof 134

5.2 Kolmogorov–Smirnov statistics 1355.3 CLT in general 139

5.3.1 Conventional proof 140






x Contents

5.3.2 Second proof 1415.3.3 Errors 143

5.4 The local limit 1445.5 Figures of merit 146

5.5.1 Gibbs’s lemma and entropy 1465.5.2 Fisher’s information 1485.5.3 Log Sobolev 148

5.6 Gauss is prime 1495.7 The general iterated log 1525.8 Sparre-Andersen’s combinatorial method 152

5.8.1 Sparre-Andersen’s combinatorial lemma 1525.8.2 Application to random walk 1545.8.3 Spitzer’s identity 155

5.9 CLT in dimensions 2 or more 1575.9.1 Gaussian variables 1575.9.2 Gauss and independence 1585.9.3 CLT itself 1595.9.4 Maxwell’s distribution 160

5.10 Measure in dimension +∞ 1625.10.1 A better way 1635.10.2 Curvature 1645.10.3 A more delicate description 165

5.11 Prime numbers 166

6 Brownian Motion 1706.1 Preview 1716.2 Direct construction of BM(1) 175

6.2.1 P. Levy’s construction 1756.2.2 Wiener’s construction 178

6.3 Markov property and passage times 1796.3.1 The simple Markov property 1806.3.2 The strict Markov property 1806.3.3 Passage times 1816.3.4 Two-sided passage or the Gambler’s Ruin 183

6.4 The invariance principle 1846.4.1 Proof 1856.4.2 Reprise 187

6.5 Volume RW(1) (reprise) 1886.6 Arcsine (reprise) 191

6.6.1 Feynman–Kac (FK) 192






Contents xi

6.6.2 Proof by Brownian paths 1946.6.3 Still another way by Brownian paths 196

6.7 Skorokhod embedding 1996.7.1 CLT 2006.7.2 The iterated log 201

6.8 Kolmogorov–Smirnov (reprise) 2026.8.1 Tied Brownian motion 2036.8.2 Tied Poisson walk 2046.8.3 Evaluations 205

6.9 Ito’s lemma 2066.9.1 Brownian integrals and differentials 2076.9.2 An example 2076.9.3 Ito’s lemma 2086.9.4 Robert Brown and Einstein 213

6.10 Brownian motion in dimensions ≥ 2 2146.10.1 Ito’s lemma 2146.10.2 BM(2): some details 2156.10.3 BM(3) and how it goes to ∞ 217

6.11 S∞(√∞) revisited 221

6.11.1 div, grad, and all that 2216.11.2 Hermite and polynomial chaos 2236.11.3 The Brownian format 2246.11.4 Back to Δ 2266.11.5 Drift and Jacobian 228

7 Markov Chains 2317.1 Set-up and the Markov property 2327.2 The invariant distribution 233

7.2.1 Geometrical proof 2337.2.2 Analytical proof 2357.2.3 Probabilistic proof 235

7.3 LLN for chains 2377.3.1 LLN improved 2387.3.2 Mixing 2407.3.3 McMillan’s theorem 240

7.4 CLT for chains 2417.4.1 Kubo’s formula 2437.4.2 CLT improved 244

7.5 Real time 2447.5.1 The Markov property 245






xii Contents

7.5.2 Loops and the invariant distribution 246

7.6 The standard Poisson process 248

7.7 Large deviations 250

7.7.1 Setup and simplest examples of the mainresult 250

7.7.2 Preliminaries about I 252

7.7.3 Proof of the main result 255

7.7.4 Legendre duality 257

8 The Ergodic Theorem 260

8.1 Hamiltonian mechanics 260

8.2 Gibbs, Birkhoff, and the statistical method 262

8.2.1 Gibbs’s canonical ensemble 262

8.2.2 Time averages 263

8.2.3 H. Weyl’s example 264

8.3 A more general set-up 265

8.3.1 Metric transitivity and mixing 265

8.3.2 Poincare recurrence 267

8.4 Riesz’s lemma and Garsia’s trick 268

8.5 Continued fractions 270

8.5.1 The set-up 270

8.5.2 Birkhoff applied 272

8.5.3 Proof of metric transitivity 273

8.5.4 Mixing 274

8.5.5 Information rate (McMillan’s theorem) 275

8.6 Geodesic flow 278

8.6.1 Sphere 278

8.6.2 Plane 279

8.6.3 Poincare’s half-plane 279

8.6.4 H2/Γ: the circle bundle 284

8.6.5 How continued fractions enter 285

8.6.6 CLT 288

8.6.7 Back to R2/Z2 288

8.6.8 Why H2/Γ is better 289






Contents xiii

9 Communication over a Noisy Channel 2909.1 Information/Uncertainty/Entropy 291

9.1.1 What Boltzmann said 2949.1.2 Information as a guide to gambling 2969.1.3 A dishonest coin 2979.1.4 Relative entropy 298

9.2 Noiseless coding 2999.3 The source 300

9.3.1 The rate 3019.3.2 McMillan’s theorem (reprise) 302

9.4 The noisy channel: capacity 3059.4.1 Simplest example 306

9.5 The noisy channel: coding 3089.6 Communication when H > C 3099.7 Communication when H < C 3109.8 The binary symmetric channel 313

9.8.1 Shannon’s idea for H < C 3139.8.2 Garbage out 316

10 Equilibrium Statistical Mechanics 31710.1 What Gibbs said 317

10.1.1 Phase space and energy 31710.1.2 The microcanonical ensemble 31810.1.3 Heat bath and the canonical ensemble 31910.1.4 Large volume 32010.1.5 Thermodynamics: free energy 32110.1.6 Thermodynamics: pressure 323

10.2 Two simple examples 32610.2.1 Ideal gas 32610.2.2 Hard balls 326

10.3 Van der Waals’ gas law: dimension 1 32810.4 Van der Waals: dimension 3 331

10.4.1 Z bounded below 33110.4.2 Scaling and the van der Waals limit 33310.4.3 Finishing the proof 333

10.5 The Ising model 33410.5.1 Overview 33410.5.2 Dimension 1 33610.5.3 Dimension 2 337

10.6 Existence of Z 338






xiv Contents

10.7 Magnetization per spin: dimension 2 341

10.7.1 Shape of m, T fixed 341

10.7.2 Shape of m, K fixed 342

10.8 Change of phase: dimension 2 345

10.8.1 High temperature 345

10.8.2 Low temperature 348

10.9 Duality and the critical temperature 350

11 Statistical Mechanics Out of Equilibrium 352

11.1 What Boltzmann said and what came after 354

11.2 The two-speed gas: chaos and the law of largenumbers 363

11.2.1 The empirical distribution 364

11.2.2 Chaos and the law of large numbers 365

11.2.3 Why chaos propagates 367

11.3 The two-speed gas: fluctuations 369

11.4 More about Boltzmann’s equation 371

11.5 The two-speed gas with streaming 374

11.5.1 Solving Boltzmann 375

11.5.2 Carleman’s gas 376

11.5.3 The surprising equation 377

11.5.4 Velocity and displacement 380

11.6 Chapman–Enskog–Hilbert 381

11.6.1 First pass 382

11.6.2 Second pass 383

11.6.3 Making better sense of all that 384

11.6.4 Focusing 385

11.7 Kac’s gas 385

11.7.1 Boltzmann’s equation and Wild’s sum 386

11.7.2 Entropy and the tendency to equilibrium 390

11.7.3 Fisher’s information 391

11.7.4 CLT: Trotter’s method 393

11.7.5 CLT: Grunbaum’s method 395

11.7.6 A tagged molecule 397

11.7.7 CLT: S∞(√∞) revisited 400






Contents xv

12 Random Matrices 40212.1 The Gaussian orthogonal ensemble (GOE) 40212.2 Why a semi-circle? 404

12.2.1 Reduction to spec x 40412.2.2 Steepest descent 405

12.3 The semi-circle: a hands-on proof 40812.3.1 Wiener’s recipe 40912.3.2 Traces 40912.3.3 Samples 41012.3.4 A better way 41112.3.5 Leading order 41212.3.6 Convergence of traces 41312.3.7 The semi-circle 414

12.4 Dyson’s Coulomb gas 41512.4.1 2 × 2 hands-on 41512.4.2 n× n in general 41612.4.3 Coda 418

12.5 Brownian motion without crossing 41912.5.1 Crossing times 41912.5.2 Connection to spec x 420

12.6 The Gaussian unitary ensemble 42212.6.1 Reduction to spec x 42212.6.2 Scaling and the semi-circle 42312.6.3 Dyson’s gas 423

12.7 How to compute 42412.7.1 Andreief’s lemma 42412.7.2 Application to spec x: first pass 42512.7.3 Hermite polynomials 42612.7.4 Application to spec x: second pass 42712.7.5 Fredholm determinants 42812.7.6 Application to spec x: third pass 430

12.8 In the bulk 43112.8.1 A single gap 43212.8.2 Wigner’s surmise 433

12.9 The ODE 43412.10 The tail 439

12.10.1 Wigner’s surmise (corrected) 44212.11 At the edge 44312.12 Coda 444

12.12.1 Some history old and new 444






xvi Contents

12.12.2 What’s happening 44512.12.3 Riemann and the prime numbers 445

Bibliography 447Index 458






Preface

The goal of this book is to present the elementary facts of classicalprobability, namely the law of large numbers (LLN) and the centrallimit theorem (CLT), first in the simplest setting (Bernoulli trials), thenin more generality, and finally in some of their ramifications in, e.g.arithmetic, geometry, information and coding, and classical mechanics.

Let’s talk about coin-tossing to illustrate the principal themes in thesimplest way.

Let the coin be honest so that the probability of heads or tails is thesame (= 1/2). After a large number (n) of independent trials, you have# = e1 + · · · + en successes, meaning heads, let’s say e = 1 for heads,e = 0 for tails. The law of large numbers states that #/n tends to 1/2as n ↑ ∞ with probability 1 – P

[limn↑∞

#n = 1

2

]= 1 as it is written. That’s

only common sense if you like. The central limit theorem is deeper. Itsays that if you center and scale # as in (#− n/2) over

√n/2, then for

large n you will see the celebrated bell-shaped curve of Gauss:

limn↑∞

P

[a ≤ # − n/2√

n/2< b

]=∫ b

a

e−x2/2

√2π

dx for any a < b.

I say it lies deeper, but it is only the proof that it is so. The phe-nomenon itself is easily illustrated in Nature. To do this, it is best tomake a little change, making new es from the old by the rule e →2(e− 1

2

)= ±1. Then x(n) = e1+· · ·+en is the standard random walk, so-

called, taking independent steps ±1 with probabilities P(e = ±1) = 12 ,

and you have the more symmetrical law:

limn↑∞

P

[a ≤ x(n)√

n< b

]=∫ b

a

e−x2/2

√2π

dx.

xvii






xviii Preface

Figure 1

The walk is easily simulated. Take a board studded with nails as inFigure 1 (left) and incline it not too steeply (at 20◦ say), pour in birdshot from a little funnel at the top, and look to see what piles up at thebottom. You should see a scaled approximation to Gauss’s celebratedbell-shaped curve (2π)−1/2e−x2/2 as in Figure 1 (right) and that is justwhat happens: the individual shot, running down, hits a nail and isdeflected, roughly half the time to the right and half the time to the left,imitating the random walk to produce a bell-shaped heap at the bottomin vindication of CLT.

Of course, that is not exactly what happens: the shot is not perfectlyround, the nails are not perfectly placed, perfect statistical independencedoes not exist in Nature, and so on. But it is approximately so, whichis why I have placed on the title page the maxim of La Rochefoucauld(who surely never thought about the standard random walk but has itjust right): That there are certain deceptions, wrong in fact, but whichcome so close to the real truth that it would be a mistake of judgmentnot to let oneself be fooled by them. Or, to vary the mot : Probability isonly a manner of speaking – not the real thing – but is wonderful howwell it works.

Take for example, Gibbs’s statistical mechanics of which you will geta glimpse in Chapter 10. It is based, of course, on ideas from Nature,but the language is probability, and it is successful beyond all dreams.Or again, take Shannon’s ideas about the quantity of information andthe means (coding) for its faithful communication over a noisy channel –an equally successful statistical picture of the thing, explained, in part,in Chapter 9.

Well, you get the idea of what I want to do and will judge at the endif I succeeded. Einstein said: “Everything should be made as simple aspossible, but not more simple”. I have tried to follow that.






Preface xix

AcknowledgementMy debt to Anne Boutet de Monvel is very great. She has typed thewhole book from my poor writing with such care, both for how it looksand what it says. I do not know how I could have finished it without herfriendly help.

Henry McKeanNYC and Essex, MA, December 2013






Guide

PrerequisitiesNot much. I need a good working knowledge of the vector spaces Rd andCd, and of calculus in several variables; also the rudiments of probabilityand some knowledge of Lebesgue’s measure and integral on the unitinterval [0, 1]. As to the last, I will sketch what I need and ask youto read up on it if you don’t know it already, or just to believe whatI tell you and use it (with care). Here are some books at about theright level: Munroe [1953], Breiman [1968], and/or the more advancedBillingsley [1979]. Any of these will tell what is wanted, through thelittle sketch in §1.1 is plenty if you fill it in. Besides, some complexfunction theory would be nice; it is used sparingly. Ahlfors [1979] is bestfor this. As to basic probability, Breiman [1968] is excellent and aboveall Feller [1966] which is full of verve, much information, and a varietyof practical examples. In short, only a modest technical machinery isrequired so as to keep the probability to the fore.

ExercisesPlease do these faithfully. It is the only known way to learn the tricksof the trade. Some are marked with a star (�) as being unnecessary tothe sequel and/or more difficult. Certain articles and sections are alsostarred for similar reasons.

ReferencesReferences are indicated by a name followed by the year of publicationin square brackets, as in Feller [1950: 33–37]: 1950 indicates the date ofpublication, 33–37 gives the paging. These are listed at the end. Thingslike Gauss (1800) or Jacobi (1820), with the year in parentheses, arenot precise references, only historical indications: who and very roughlywhen.

xx






Guide xxi

Notations/UsagePositive means > 0, non-negative ≥ 0, and similarly for negative andnon-positive; x+/x− is the positive/negative part of the number x; x∧yis the smaller of x and y, x∨ y the larger. The symbol means approx-imately equal, up to something really small or with a small percentageerror, as the context will indicate; � is used similarly. Z is the inte-gers, Z+ the non-negative integers, N the whole numbers (1, 2, 3, . . .),R the line, C the complex plane. X is a (sample) space; A, B, C andthe like are (mostly) events, i.e. subsets of X. The symbol F denotes afield (of events) meaning that it is closed under complements indicatedby a prime (′), countable unions (∪), and countable intersections (∩) –not the common usage but more brief. German (fraktur) is reserved forthese. Boldface x and the like is used, not wholly consistently, for ran-dom quantities. Italic x and the like is (mostly) for non-random things.P is probability. E is expectation as in E(x) =

∫xd P; the shorthand

E(x, A) is for∫

Axd P. By C[0, 1], C2[0,∞), C∞(R), are designated vari-

ous classes of continuous functions. C∞↓ (R) is the class of smooth, rapidly

vanishing functions. L1(R), L2(R), and so on are the usual Lebesguespaces. # is for counting as in #(p ≤ n : p a prime number). 2+ meansa number a little bit more than 2, say; 2− means a number a littlebit less. The natural logarithm to the base e = 2.718+ is written log;loglog means log(log). The symbol 1K is an indicator function: 1 on K,0 elsewhere. Traces are denoted by tr.

For those not so familiar with spherical polar coordinates in threedimensions, I remind you that x ∈ R3 may be written x = |x|e in which|x| is length and e is the (unit) direction (sinφ cos θ, sinφ sin θ, cosφ).Here x3 = |x| cosφ, −π

2 ≤ φ ≤ π2 being co-latitude, and 0 ≤ θ ≤ 2π is

the longitude, measured counter-clockwise in the plane x3 = 0 as in thepicture.

x

φ

θ






probability: the classical limit theoremsassets.cambridge.org/97811070/53212/frontmatter/... ·...

Documents