probability random processes and ergodic properties
Post on 08-Dec-2016
Embed Size (px)
Probability, Random Processes,and Ergodic Properties
Robert M. Gray
Probability, Random Processes,and Ergodic Properties
10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connectionwith any form of information storage and retrieval, electronic adaptation, computer software, or by similaror dissimilar methodology now known or hereafter developed is forbidden.The use in this publication of trade names, trademarks, service marks, and similar terms, even if they arenot identified as such, is not to be taken as an expression of opinion as to whether or not they are subjectto proprietary rights.
Printed on acid-free paper
This work may not be translated or copied in whole or in part without the written
Springer Dordrecht Heidelberg London New York
Springer Science+Business Media, LLC 2009
Springer is part of Springer Science+Business Media (www.springer.com)
All rights reserved.permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY
Robert M. GrayDepartment of Electrical EngineeringStanford University
ISBN 978-1-4419-1089-9 e-ISBN 978-1-4419-1090-5DOI 10.1007/978-1-4419-1090-5
Library of Congress Control Number: 2009932896
Stanford, CA 94305-4075
To my brother, Peter R. GrayApril 1940 May 2009
This book has a long history. It began over two decades ago as the firsthalf of a book then in progress on information and ergodic theory. Theintent was and remains to provide a reasonably self-contained advanced(at least for engineers) treatment of measure theory, probability theory,and random processes, with an emphasis on general alphabets and onergodic and stationary properties of random processes that might be nei-ther ergodic nor stationary. The intended audience was mathematicallyinclined engineering graduate students and visiting scholars who hadnot had formal courses in measure theoretic probability or ergodic the-ory. Much of the material is familiar stuff for mathematicians, but manyof the topics and results had not then previously appeared in books. Sev-eral topics still find little mention in the more applied literature, includ-ing descriptions of random processes as dynamical systems or flows;general alphabet models including standard spaces, Polish spaces, andLebesgue spaces; nonstationary and nonergodic processes which havestationary and ergodic properties; metrics on random processes whichquantify how different they are in ways useful for coding and signal pro-cessing; and stationary or sliding-block mappings coding or filtering of random processes.
The original project grew too large and the first part contained muchthat would likely bore mathematicians and discourage them from thesecond part. Hence I followed a suggestion to separate the materialand split the project in two. The original justification for the presentmanuscript was the pragmatic one that it would be a shame to waste allthe effort thus far expended. A more idealistic motivation was that thepresentation had merit as filling a unique, albeit small, hole in the litera-ture. Personal experience indicates that the intended audience rarely hasthe time to take a complete course in measure and probability theoryin a mathematics or statistics department, at least not before they needsome of the material in their research.
Many of the existing mathematical texts on the subject are hard forthe intended audience to follow, and the emphasis is not well matchedto engineering applications. A notable exception is Ashs excellent text, which was likely influenced by his original training as an electricalengineer. Still, even that text devotes little effort to ergodic theorems,perhaps the most fundamentally important family of results for apply-ing probability theory to real problems. In addition, there are many otherspecial topics that are given little space (or none at all) in most texts onadvanced probability and random processes. Examples of topics devel-oped in more depth here than in most existing texts are the following.
Random processes with standard alphabets
The theory of standard spaces as a model of quite general process alpha-bets is developed in detail. Although not as general (or abstract) as ex-amples sometimes considered by probability theorists, standard spaceshave useful structural properties that simplify the proofs of some gen-eral results and yield additional results that may not hold in the moregeneral abstract case. Three important examples of results holding forstandard alphabets that have not been proved in the general abstractcase are the Kolmogorov extension theorem, the ergodic decomposition,and the existence of regular conditional probabilities. All three resultsplay fundamental roles in the development and understanding of math-ematical information theory and signal processing. Standard spaces in-clude the common models of finite alphabets (digital processes), count-able alphabets, and real alphabets as well as more general completeseparable metric spaces (Polish spaces). Thus they include many func-tion spaces, Euclidean vector spaces, two-dimensional image intensityrasters, etc. The basic theory of standard Borel spaces may be found inthe elegant text of Parthasarathy , and treatments of standard spacesand the related Lusin and Suslin spaces may be found in Blackwell ,Christensen , Schwartz , Bourbaki , and Cohn .
The development of the basic results here is more coding-oriented andattempts to separate clearly the properties of standard spaces, whichare useful and easy to manipulate, from the demonstrations that certainspaces are standard, which are more complicated and can be skipped.Unlike the traditional treatments, we define and study standard spacesfirst from a purely probability theory point of view and postpone thetopological metric space details until later. In a nutshell, a standardmeasurable space is one for which any nonnegative set function that isfinitely additive (usually easy to prove) on a field has a unique extensionto a probability measure on the -field generated by the field. Spaceswith this property ease the construction of probability measures fromset functions that arise naturally in real systems, such as sample aver-
ages and relative frequencies. The fact that well behaved (separable andcomplete) metric spaces with the usual (Borel) -field are standard alsoprovides a natural measure of approximation (or distortion) betweenrandom variables, vectors, and processes that will be seen to be quiteuseful in quantifying the performance of codes and filters. This bookand its sequel have numerous overlaps with ergodic theory, and mostof the ergodic theory literature focuses on another class of probabilityspaces and the associated models of random processes and dynamicalspaces Lebesgue spaces. Standard spaces and Lebesgue spaces are in-timately connected, and both are discussed here and comparisons made.The full connection between the two is stated, but not proved becausedoing so would require significant additional material not part of theprimary goals of this book.
Nonstationary and nonergodic processes
The theory of asymptotically mean stationary processes and the ergodicdecomposition are treated in depth in order to model many physical pro-cesses better than can traditional stationary and ergodic processes. Bothtopics are virtually absent in all books on random processes, yet they arefundamental to understanding the limiting behavior of nonergodic andnonstationary processes. Both topics are considered in Krengels excel-lent book on ergodic theorems , but the treatment here is in greaterdepth. The treatment considers both the common two-sided or bilateralrandom processes, which are considered to have been producing outputsforever, and the often more difficult one-sided or unilateral processes,which better model processes that are turned on at some specific timeand which exhibit transient behavior,.
Ergodic properties and theorems
The notions of time averages and probabilistic averages are developedtogether in order to emphasize their similarity and to demonstrate manyof the implications of the existence of limiting sample averages. The er-godic theorem is proved for the general case of asymptotically meanstationary processes. In fact, it is shown that asymptotic mean stationar-ity is both sufficient and necessary for the classical pointwise or almosteverywhere ergodic theorem to hold for all bounded measurements. Thesubadditive ergodic theorem of Kingman  is included as it is usefulfor studying the limiting behavior of certain measurements on randomprocesses that are not simple arithmetic averages. The proofs are basedon simple proofs of the ergodic theorem developed by Ornstein andWeiss , Katznelson and Weiss , Jones , and Shields . These
proofs use coding arguments reminiscent of information and communi-cation theory rather than the traditional (and somewhat tricky) maximalergodic theorem. The interrelations of stationary and ergodic propertiesof processes that are stationary or ergodic with respect to block shiftsare derived. Such processes arise naturally in coding and signal process-ing systems that operate sequentially on disjoint blocks of values pro-duced by stationary and ergodic processes. The topic largely originatedwith Nedoma  and it plays an important role in the general versionsof Shannon channel and source coding theorems in the sequel and else-where.
Process distance measures
Measures of distance between random processes which quantify howclose one process is to another are deve