Stochastic Processes Ross

Download Stochastic Processes Ross

Post on 26-Oct-2014




52 download

Embed Size (px)


<p>STOCHASTIC PROCESSESSecond Edition</p> <p>Sheldon M. RossUniversity of California, Berkeley</p> <p>JOHN WILEY &amp; SONS. INC. New York Chichester Brisbane Toronto Singapore</p> <p>ACQUISITIONS EDITOR Brad Wiley II MARKETING MANAGER Debra Riegert SENIOR PRODUCfION EDITOR Tony VenGraitis MANUFACfURING MANAGER Dorothy Sinclair TEXT AND COVER DESIGN A Good Thing, Inc PRODUCfION COORDINATION Elm Street Publishing Services, Inc This book was set in Times Roman by Bi-Comp, Inc and printed and bound by Courier/Stoughton The cover was printed by Phoenix Color Recognizing the importance of preserving what has been written, it is a policy of John Wiley &amp; Sons, Inc to have books of enduring value published in the United States printed on acid-free paper, and we exert our best efforts to that end The paper in this book was manufactured by a mill whose forest management programs include sustained yield harvesting of its timberlands Sustained yield harvesting principles ensure that the number of trees cut each year does not exceed the amount of new growth Copyright 1996, by John Wiley &amp; Sons, Inc All rights reserved Published simultaneously in Canada Reproduction or translation of any part of this work beyond that permitted by Sections 107 and 108 of the 1976 United States Copyright Act without the permission of the copyright owner is unlawful Requests for permission or further information shOUld be addressed to the Permissions Department, John Wiley &amp; Sons, Inc</p> <p>Library of Congress Cataloging-in-Publication Data: Ross, Sheldon M Stochastic processes/Sheldon M Ross -2nd ed p cm Includes bibliographical references and index ISBN 0-471-12062-6 (cloth alk paper) 1 Stochastic processes I Title QA274 R65 1996 5192-dc20Printed in the United States of America 10 9 8 7 6 5 4 3 2</p> <p>95-38012 CIP</p> <p>On March 30, 1980, a beautiful six-year-old girl died. This book is dedicated to the memory of</p> <p>Nichole Pomaras</p> <p>Preface to the First Edition</p> <p>This text is a nonmeasure theoretic introduction to stochastic processes, and as such assumes a knowledge of calculus and elementary probability_ In it we attempt to present some of the theory of stochastic processes, to indicate its diverse range of applications, and also to give the student some probabilistic intuition and insight in thinking about problems We have attempted, wherever possible, to view processes from a probabilistic instead of an analytic point of view. This attempt, for instance, has led us to study most processes from a sample path point of view. I would like to thank Mark Brown, Cyrus Derman, Shun-Chen Niu, Michael Pinedo, and Zvi Schechner for their helpful commentsSHELDON</p> <p>M. Ross</p> <p>vii</p> <p>Preface to the Second Edition</p> <p>The second edition of Stochastic Processes includes the following changes' (i) Additional material in Chapter 2 on compound Poisson random variables, including an identity that can be used to efficiently compute moments, and which leads to an elegant recursive equation for the probabilIty mass function of a nonnegative integer valued compound Poisson random variable; (ii) A separate chapter (Chapter 6) on martingales, including sections on the Azuma inequality; and (iii) A new chapter (Chapter 10) on Poisson approximations, including both the Stein-Chen method for bounding the error of these approximations and a method for improving the approximation itself. In addition, we have added numerous exercises and problems throughout the text. Additions to individual chapters follow: In Chapter 1, we have new examples on the probabilistic method, the multivanate normal distnbution, random walks on graphs, and the complete match problem Also, we have new sections on probability inequalities (including Chernoff bounds) and on Bayes estimators (showing that they are almost never unbiased). A proof of the strong law of large numbers is given in the Appendix to this chapter. New examples on patterns and on memoryless optimal coin tossing strategies are given in Chapter 3. There is new matenal in Chapter 4 covering the mean time spent in transient states, as well as examples relating to the Gibb's sampler, the Metropolis algonthm, and the mean cover time in star graphs. Chapter 5 includes an example on a two-sex population growth model. Chapter 6 has additional examples illustrating the use of the martingale stopping theorem. Chapter 7 includes new material on Spitzer's identity and using it to compute mean delays in single-server queues with gamma-distnbuted interarrival and service times. Chapter 8 on Brownian motion has been moved to follow the chapter on martingales to allow us to utilIZe martingales to analyze Brownian motion.ix</p> <p>x</p> <p>PREFACE TO THE SECOND EDITION</p> <p>Chapter 9 on stochastic order relations now includes a section on associated random variables, as well as new examples utilizing coupling in coupon collecting and bin packing problems. We would like to thank all those who were kind enough to write and send comments about the first edition, with particular thanks to He Sheng-wu, Stephen Herschkorn, Robert Kertz, James Matis, Erol Pekoz, Maria Rieders, and Tomasz Rolski for their many helpful comments.SHELDON</p> <p>M. Ross</p> <p>Contents</p> <p>CHAPTER</p> <p>1. PRELIMINARIES1.1. 1.2. 1.3. 1.4 1.5.</p> <p>1</p> <p>1.6. 1.7. 1.8. 1.9.</p> <p>Probability 1 Random Variables 7 Expected Value 9 Moment Generating, Characteristic Functions, and 15 Laplace Transforms Conditional ExpeCtation 20 1.5.1 Conditional Expectations and Bayes 33 Estimators The Exponential Distribution, Lack of Memory, and 35 Hazard Rate Functions Some Probability Inequalities 39 Limit Theorems 41 Stochastic Processes 41 Problems References Appendix 46 55 56</p> <p>CHAPTER 2. THE POISSON PROCESS</p> <p>5964 66</p> <p>2 1. The Poisson Process 59 22. Interarrival and Waiting Time Distributions 2.3 Conditional Distribution of the Arrival Times 2.31. The MIGII Busy Period 73 2.4. Nonhomogeneous Poisson Process 78 2.5 Compound Poisson Random Variables and 82 Processes 2.5 1. A Compound Poisson Identity 84 25.2. Compound Poisson Processes 87</p> <p>xi</p> <p>xii2.6 Conditional Poisson Processes Problems References 89 97 88</p> <p>CONTENTS</p> <p>CHAPTER</p> <p>3. RENEWAL THEORY</p> <p>98</p> <p>3.1 Introduction and Preliminaries 98 32 Distribution of N(t) 99 3 3 Some Limit Theorems 101 331 Wald's Equation 104 332 Back to Renewal Theory 106 34 The Key Renewal Theorem and Applications 109 341 Alternating Renewal Processes 114 342 Limiting Mean Excess and Expansion of met) 119 343 Age-Dependent Branching Processes 121 3.5 Delayed Renewal Processes 123 3 6 Renewal Reward Processes 132 3 6 1 A Queueing Application 138 3.7. Regenerative Processes 140 3.7.1 The Symmetric Random Walk and the Arc Sine Laws 142 3.8 Stationary Point Processes 149 Problems References 153 161</p> <p>CHAPTER</p> <p>4. MARKOV CHAINS</p> <p>163</p> <p>41 Introduction and Examples 163 42. Chapman-Kolmogorov Equations and Classification of States 167 4 3 Limit Theorems 173 44. Transitions among Classes, the Gambler's Ruin Problem, and Mean Times in Transient States 185 4 5 Branching Processes 191 46. Applications of Markov Chains 193 4 6 1 A Markov Chain Model of Algorithmic 193 Efficiency 462 An Application to Runs-A Markov Chain with a 195 Continuous State Space 463 List Ordering Rules-Optimality of the Transposition Rule 198</p> <p>CONTENTS</p> <p>XIII</p> <p>A </p> <p>4.7 Time-Reversible Markov Chains 48 Semi-Markov Processes 213 Problems References 219 230</p> <p>203</p> <p>CHAPTER</p> <p>5. CONTINUOUS-TIME MARKOV CHAINS</p> <p>231</p> <p>5 1 Introduction 231 52. Continuous-Time Markov Chains 231 5.3.-Birth and Death Processes 233 5.4 The Kolmogorov Differential Equations 239 5.4.1 Computing the Transition Probabilities 249 251 5.5. Limiting Probab~lities 5 6. Time Reversibility 257 5.6.1 Tandem Queues 262 5.62 A Stochastic Population Model 263 5.7 Applications of the Reversed Chain to Queueing Theory 270 271 57.1. Network of Queues 57 2. The Erlang Loss Formula 275 573 The MIG/1 Shared Processor System 278 58. Uniformization 282 Problems References</p> <p>2R6 294</p> <p>CHAPTER 6. MARTINGALES</p> <p>295</p> <p>61 62 6 3. 64. 65.</p> <p>Introduction 295 Martingales 295 Stopping Times 298 Azuma's Inequality for Martingales 305 Submartingales, Supermartingales. and the Martingale 313 Convergence Theorem A Generalized Azuma Inequality 319 Problems References 322 327</p> <p>CHAPTER 7. RANDOM WALKS</p> <p>328</p> <p>Introduction 32R 7 1. Duality in Random Walks</p> <p>329</p> <p>xiv</p> <p>CONTENTS</p> <p>7.2 Some Remarks Concerning Exchangeable Random Variables 338 73 Using Martingales to Analyze Random Walks 341 74 Applications to GI Gil Queues and Ruin 344 Problems 7.4.1 The GIGll Queue 344 7 4 2 A Ruin Problem 347 7 5 Blackwell's Theorem on the Line 349 Problems References 352 355</p> <p>CHAPTER 8. BROWNIAN MOTION AND OTHER MARKOV 356 PROCESSES 8.1 Introduction and Preliminaries 356 8.2. Hitting Times, Maximum Variable, and Arc Sine Laws 363 83. Variations on Brownian Motion 366 83.1 Brownian Motion Absorbed at a Value 366 8.3.2 Brownian Motion Reflected at the Origin 368 8 3 3 Geometric Brownian Motion 368 8.3.4 Integrated Brownian Motion 369 8.4 Brownian Motion with Drift 372 84.1 Using Martingales to Analyze Brownian Motion 381 85 Backward and Forward Diffusion Equations 383 8.6 Applications of the Kolmogorov Equations to Obtaining Limiting Distributions 385 8.61. Semi-Markov Processes 385 862. The MIG/1 Queue 388 8.6.3. A Ruin Problem in Risk Theory 392 87. A Markov Shot Noise Process' 393 88 Stationary Processes 396 Problems References 399 403</p> <p>CHAPTER 9. STOCHASTIC ORDER RELATIONS Introduction 404 9 1 Stochastically Larger 404</p> <p>404</p> <p>CONTENTS</p> <p>xv9 2. Coupling 409 9.2 1. Stochastic Monotonicity Properties of Birth and Death Processes 416 9.2.2 Exponential Convergence in Markov 418 Chains</p> <p>0.3. Hazard Rate Ordering and Applications to Counting Processes 420 94. Likelihood Ratio Ordering 428 95. Stochastically More Variable 433 9.6 Applications of Variability Orderings 437 9.6.1. Comparison of GIGl1 Queues 439 9.6.2. A Renewal Process Application 440 963. A Branching Process Application 443 9.7. Associated Random Variables 446 Probl ems References449 456</p> <p>CHAPTER 10. POISSON APPROXIMATIONS</p> <p>457</p> <p>Introduction 457 457 10.1 Brun's Sieve 10.2 The Stein-Chen Method for Bounding the Error of the Poisson Approximation 462 10.3. Improving the Poisson Approximation 467 Problems References470 472</p> <p>ANSWERS AND SOLUTIONS TO SELECTED PROBLEMSINDEX</p> <p>473</p> <p>505</p> <p>CHAPTER</p> <p>1</p> <p>Preliminaries</p> <p>1. 1</p> <p>PROBABILITY</p> <p>A basic notion in probability theory is random experiment an experiment whose outcome cannot be determined in advance. The set of all possible outcomes of an experiment is called the sample space of that experiment, and we denote it by S. An event is a subset of a sample space, and is said to occur if the outcome of the experiment is an element of that subset. We shall suppose that for each event E of the sample space S a number P(E) is defined and satisfies the following three axioms*: Axiom (1) O:s;; P(E) ~ 1. Axiom (2) P(S) = 1 Axiom (3) For any sequence of events E., E 2 , exclusive, that is, events for which E,E, cf&gt; is the null set),P(</p> <p>that are mutually</p> <p>= cf&gt; when i ~ j (where</p> <p>9</p> <p>E,) =</p> <p>~ P(E,).</p> <p>We refer to P(E) as the probability of the event E. Some simple consequences of axioms (1). (2). and (3) are.</p> <p>1.1.1. 1.1.2. 1.1.3. 1.1.4.</p> <p>If E C F, then P(E) :S P(F). P(EC) = 1 - P(E) where EC is the complement of E.P(U~ E,) = ~~ P(E,) when the E, are mutually exclusive. P(U~ E,) :S ~~ P(E,).</p> <p>The inequality (114) is known as Boole's inequality* Actually P(E) will only be defined for the so-called measurable events of S But this restriction need not concern us</p> <p>1</p> <p>2</p> <p>PRELIMIN ARIES</p> <p>An important property of the probability function P is that it is continuous. To make this more precise, we need the concept of a limiting event, which we define as follows A sequence of even ts {E" , n 2= I} is said to be an increasing sequence if E" C E,,+1&gt; n ;&gt; 1 and is said to be decreasing if E" :J E,,+I, n ;&gt; 1. If {E", n 2= I} is an increasing sequence of events, then we define a new event, denoted by lim"_,,, E" byOIl</p> <p>limE,,= ,,_00</p> <p>U,"'I</p> <p>E;</p> <p>when E" C E,,+I, n 2= 1.</p> <p>Similarly if {E", n</p> <p>2=</p> <p>I} is a decreasing sequence, then define lim,,_oo E" by</p> <p>limE" ,,_00</p> <p>=</p> <p>nOIl</p> <p>E"</p> <p>when E" :J E,,+I, n 2= 1.</p> <p>We may now state the following:</p> <p>PROPOSITION 1.1.1If {E", n2:</p> <p>1} is either an increasing or decreasing sequence of events, then</p> <p>Proof Suppose, first, that {E", n 2: 1} is an increasing sequence, and define events F", n 2: 1 by</p> <p>F"</p> <p>= E"</p> <p>(</p> <p>YII-I</p> <p>)t</p> <p>E,</p> <p>= E"E~_\,</p> <p>n&gt;l</p> <p>That is, F" consists of those points in E" that are not in any of the earlier E,. i &lt; n It is easy to venfy that the Fn are mutually exclusive events such that</p> <p>'" U F, = U E, ,=\ ,=1</p> <p>and</p> <p>" U F, " E, =U ,=\i-I</p> <p>for all n</p> <p>2:</p> <p>1.</p> <p>PROBABILITY</p> <p>;3</p> <p>Thus</p> <p>p(y E,) =p(y F.)=</p> <p>" L P(F,)I</p> <p>(by Axiom 3)</p> <p>n</p> <p>= n_7J limLP(F,)I</p> <p>= n_:r:l limp(U 1</p> <p>F.)</p> <p>= n_7J limp(UI</p> <p>E,)</p> <p>= lim PeEn),n~"</p> <p>which proves the result when {En' n ;:::: I} is increasing If {En. n ;:::: I} is a decreasing sequence, then {E~, n ;:::: I} is an increasing sequence, hence,</p> <p>P</p> <p>(0 E~) = lim P(E~)1n_:o</p> <p>But, as U~ F;~</p> <p>=</p> <p>(n~ En)', we see that</p> <p>1- Por, equivalently,</p> <p>(01</p> <p>En) =</p> <p>~~"2l1 - PeEn)],</p> <p>Pwhich proves the result</p> <p>(n En) = lim PeEn),"_00</p> <p>EXAMPLE 1..1.(A) Consider a population consisting of individuals able to produce offspring of the same kind. The number of individuals</p> <p>4</p> <p>PRELIMINARIES</p> <p>initially present, denoted by Xo. is called the size of the zeroth generation All offspring of the zeroth generation constitute the first generation and their number is denoted by Xl In general, let Xn denote the size of the nth generation Since Xn = 0 implies that X n+ l = 0, it follows that P{Xn = O} is increasing and thus limn _ oc P{Xn = O} exists What does it represent? To answer this use Proposition 1.1.1 as foHows:</p> <p>= P{the population ever dies out}.That is, the limiting probability that the nth generation is void of individuals is equal to the probability of eventual extinction of the population. Proposition 1.1.1 can also be used to prove the Borel-Cantelli lemma.</p> <p>PROPOSITION 1...1.2The Borel-Cantelli Lemma Let EI&gt; E 2 , denote a sequence of events If"" L: P(E,) O</p> <p>PRELIMINARIES</p> <p>if N = 0, then (1.3.5) and (1.3.6) yield</p> <p>1- I or (1.3.7)</p> <p>=~</p> <p>(~) (-1)'</p> <p>1=</p> <p>~ (~) (-1)1+1</p> <p>Taking expectations of both sides of (1.3.7) yields(1.3.8) E[Il</p> <p>= E[N]</p> <p>- E [(:)]</p> <p>+ ... + (-1)n+IE [(:)].</p> <p>However,E[I]</p> <p>= P{N&gt; O}== P{at least one of the A, occurs}</p> <p>andE[N] = E</p> <p>[~~] = ~ P(A,),</p> <p>E</p> <p>[(~)] = E[number of pairs of the A, that occur]</p> <p>=</p> <p>LL P(A,A), 1 0, b &gt; 0</p> <p>cx</p> <p>1(1</p> <p>C-</p> <p>_ rea + b) f(a)f(b)</p> <p>a a+b</p> <p>+ b)Z(a + b + I)</p> <p>MOMENT GENERATING. CHARACfERISTIC FUNCfIONS. AND LAPLACE</p> <p>,19</p> <p>then the random variables XI, ate normal distribution. Let us now consider</p> <p>., Xm...</p>