probability, statistics, and random processes for engineers · “a01_star1236_04_se_fm” —...

“A01_STAR1236_04_SE_FM” — 2011/7/8 — 19:34 — page i

Probability, Statistics,and Random Processesfor Engineers

Fourth Edition

Henry Stark

Illinois Institute of Technology

John W. WoodsRensselaer Polytechnic Institute

Boston Columbus Indianapolis New York San Francisco Upper Saddle RiverAmsterdam Cape Town Dubai London Madrid Milan Munich Paris Montreal TorontoDelhi Mexico City Sao Paulo Sydney Hong Kong Seoul Singapore Taipei Tokyo

©2012 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved.

“A01_STAR1236_04_SE_FM” — 2011/7/8 — 19:34 — page ii

To my father P.D. Stark (in memoriam)From darkness to light (1941–1945)

Henry Stark

To Harriet.

John. W. Woods

Vice President and Editorial Director, ECS:Marcia J. Horton

Senior Editor: Andrew GilfillanAssociate Editor: Alice DworkinEditorial Assistant: William OpaluchMarketing Manager: Tim GalliganSenior Managing Editor: Scott Disanno

Operations Specialist: Lisa McDowellProduction Liason: Irwin ZuckerArt Director: Jayne ConteCover Designer: Bruce KenselaarFull-Service Project Management/

Composition: Integra SoftwareServices Pvt. Ltd

Copyright c© 2012, 2002, 1994, 1986 by Pearson Education, Inc., publishing as Prentice Hall. Allrights reserved. Manufactured in the United States of America. This publication is protected by Copyright,and permission should be obtained from the publisher prior to any prohibited reproduction, storage ina retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying,recording, or likewise. To obtain permission(s) to use material from this work, please submit a writtenrequest to Pearson Education, Inc., Permissions Department, One Lake Street, Upper Saddle River, NewJersey 07458, or you may fax your request to 201-236-3290.

Many of the designations by manufacturers and seller to distinguish their products are claimed as trade-marks. Where those designations appear in this book, and the publisher was aware of a trademark claim,the designations have been printed in initial caps or all caps.

The author and publisher of this book have used their best efforts in preparing this book. These effortsinclude the development, research, and testing of the theories and programs to determine their effectiveness.The author and publisher make no warranty of any kind, expressed or implied, with regard to these programsor the documentation contained in this book. The author and publisher shall not be liable in any event forincidental or consequential damages in connection with, or arising out of, the furnishing, performance, oruse of these programs.

Library of Congress Cataloging-in-Publication Data on File

10 9 8 7 6 5 4 3 2 1

ISBN 10: 0-13-231123-2ISBN 13: 978-0-13-231123-6


“A01_STAR1236_04_SE_FM” — 2011/7/8 — 19:34 — page iii

Contents

Preface xi

1 Introduction to Probability 11.1 Introduction: Why Study Probability? 11.2 The Different Kinds of Probability 2

Probability as Intuition 2Probability as the Ratio of Favorable to Total Outcomes (Classical Theory) 3Probability as a Measure of Frequency of Occurrence 4Probability Based on an Axiomatic Theory 5

1.3 Misuses, Miscalculations, and Paradoxes in Probability 71.4 Sets, Fields, and Events 8

Examples of Sample Spaces 81.5 Axiomatic Definition of Probability 151.6 Joint, Conditional, and Total Probabilities; Independence 20

Compound Experiments 231.7 Bayes’ Theorem and Applications 351.8 Combinatorics 38

Occupancy Problems 42Extensions and Applications 46

1.9 Bernoulli Trials—Binomial and Multinomial Probability Laws 48Multinomial Probability Law 54

1.10 Asymptotic Behavior of the Binomial Law: The Poisson Law 571.11 Normal Approximation to the Binomial Law 63Summary 65Problems 66References 77

iii©2012 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved.

“A01_STAR1236_04_SE_FM” — 2011/7/8 — 19:34 — page iv

iv Contents

2 Random Variables 792.1 Introduction 792.2 Definition of a Random Variable 802.3 Cumulative Distribution Function 83

Properties of FX(x) 84Computation of FX(x) 85

2.4 Probability Density Function (pdf) 88Four Other Common Density Functions 95More Advanced Density Functions 97

2.5 Continuous, Discrete, and Mixed Random Variables 100Some Common Discrete Random Variables 102

2.6 Conditional and Joint Distributions and Densities 107Properties of Joint CDF FXY (x, y) 118

2.7 Failure Rates 137Summary 141Problems 141References 149Additional Reading 149

3 Functions of Random Variables 1513.1 Introduction 151

Functions of a Random Variable (FRV): Several Views 1543.2 Solving Problems of the Type Y = g(X) 155

General Formula of Determining the pdf of Y = g(X) 1663.3 Solving Problems of the Type Z = g(X,Y ) 1713.4 Solving Problems of the Type V = g(X,Y ), W = h(X,Y ) 193

Fundamental Problem 193Obtaining fV W Directly from fXY 196

3.5 Additional Examples 200Summary 205Problems 206References 214Additional Reading 214

4 Expectation and Moments 2154.1 Expected Value of a Random Variable 215

On the Validity of Equation 4.1-8 2184.2 Conditional Expectations 232

Conditional Expectation as a Random Variable 2394.3 Moments of Random Variables 242

Joint Moments 246Properties of Uncorrelated Random Variables 248Jointly Gaussian Random Variables 251

4.4 Chebyshev and Schwarz Inequalities 255


“A01_STAR1236_04_SE_FM” — 2011/7/8 — 19:34 — page v

Contents v

Markov Inequality 257The Schwarz Inequality 258

4.5 Moment-Generating Functions 2614.6 Chernoff Bound 2644.7 Characteristic Functions 266

Joint Characteristic Functions 273The Central Limit Theorem 276

4.8 Additional Examples 281Summary 283Problems 284References 293Additional Reading 294

5 Random Vectors 2955.1 Joint Distribution and Densities 2955.2 Multiple Transformation of Random Variables 2995.3 Ordered Random Variables 302

Distribution of area random variables 3055.4 Expectation Vectors and Covariance Matrices 3115.5 Properties of Covariance Matrices 314

Whitening Transformation 3185.6 The Multidimensional Gaussian (Normal) Law 3195.7 Characteristic Functions of Random Vectors 328

Properties of CF of Random Vectors 330The Characteristic Function of the Gaussian (Normal) Law 331

Summary 332Problems 333References 339Additional Reading 339

6 Statistics: Part 1 Parameter Estimation 3406.1 Introduction 340

Independent, Identically Distributed (i.i.d.) Observations 341Estimation of Probabilities 343

6.2 Estimators 3466.3 Estimation of the Mean 348

Properties of the Mean-Estimator Function (MEF) 349Procedure for Getting a δ-confidence Interval on the Mean of a Normal

Random Variable When σX Is Known 352Confidence Interval for the Mean of a Normal Distribution When σX Is Not

Known 352Procedure for Getting a δ-Confidence Interval Based on n Observations on

the Mean of a Normal Random Variable when σX Is Not Known 355Interpretation of the Confidence Interval 355


“A01_STAR1236_04_SE_FM” — 2011/7/8 — 19:34 — page vi

vi Contents

6.4 Estimation of the Variance and Covariance 355Confidence Interval for the Variance of a Normal Random

variable 357Estimating the Standard Deviation Directly 359Estimating the covariance 360

6.5 Simultaneous Estimation of Mean and Variance 3616.6 Estimation of Non-Gaussian Parameters from Large Samples 3636.7 Maximum Likelihood Estimators 3656.8 Ordering, more on Percentiles, Parametric Versus Nonparametric Statistics 369

The Median of a Population Versus Its Mean 371Parametric versus Nonparametric Statistics 372Confidence Interval on the Percentile 373Confidence Interval for the Median When n Is Large 375

6.9 Estimation of Vector Means and Covariance Matrices 376Estimation of μ 377Estimation of the covariance K 378

6.10 Linear Estimation of Vector Parameters 380Summary 384Problems 384References 388Additional Reading 389

7 Statistics: Part 2 Hypothesis Testing 3907.1 Bayesian Decision Theory 3917.2 Likelihood Ratio Test 3967.3 Composite Hypotheses 402

Generalized Likelihood Ratio Test (GLRT) 403How Do We Test for the Equality of Means of Two Populations? 408Testing for the Equality of Variances for Normal Populations:

The F-test 412Testing Whether the Variance of a Normal Population Has a

Predetermined Value: 4167.4 Goodness of Fit 4177.5 Ordering, Percentiles, and Rank 423

How Ordering is Useful in Estimating Percentiles and the Median 425Confidence Interval for the Median When n Is Large 428Distribution-free Hypothesis Testing: Testing If Two Population are the

Same Using Runs 429Ranking Test for Sameness of Two Populations 432

Summary 433Problems 433References 439


“A01_STAR1236_04_SE_FM” — 2011/7/8 — 19:34 — page vii

Contents vii

8 Random Sequences 4418.1 Basic Concepts 442

Infinite-length Bernoulli Trials 447Continuity of Probability Measure 452Statistical Specification of a Random Sequence 454

8.2 Basic Principles of Discrete-Time Linear Systems 4718.3 Random Sequences and Linear Systems 4778.4 WSS Random Sequences 486

Power Spectral Density 489Interpretation of the psd 490Synthesis of Random Sequences and Discrete-Time Simulation 493Decimation 496Interpolation 497

8.5 Markov Random Sequences 500ARMA Models 503Markov Chains 504

8.6 Vector Random Sequences and State Equations 5118.7 Convergence of Random Sequences 5138.8 Laws of Large Numbers 521Summary 526Problems 526References 541

9 Random Processes 5439.1 Basic Definitions 5449.2 Some Important Random Processes 548

Asynchronous Binary Signaling 548Poisson Counting Process 550Alternative Derivation of Poisson Process 555Random Telegraph Signal 557Digital Modulation Using Phase-Shift Keying 558Wiener Process or Brownian Motion 560Markov Random Processes 563Birth–Death Markov Chains 567Chapman–Kolmogorov Equations 571Random Process Generated from Random Sequences 572

9.3 Continuous-Time Linear Systems with Random Inputs 572White Noise 577

9.4 Some Useful Classifications of Random Processes 578Stationarity 579

9.5 Wide-Sense Stationary Processes and LSI Systems 581Wide-Sense Stationary Case 582Power Spectral Density 584An Interpretation of the psd 586


“A01_STAR1236_04_SE_FM” — 2011/7/8 — 19:34 — page viii

viii Contents

More on White Noise 590Stationary Processes and Differential Equations 596

9.6 Periodic and Cyclostationary Processes 6009.7 Vector Processes and State Equations 606

State Equations 608Summary 611Problems 611References 633

Chapters 10 and 11 are available as Web chapters on the companionWeb site at http://www.pearsonhighered.com/stark.

10 Advanced Topics in Random Processes 63510.1 Mean-Square (m.s.) Calculus 635

Stochastic Continuity and Derivatives [10-1] 635Further Results on m.s. Convergence [10-1] 645

10.2 Mean-Square Stochastic Integrals 65010.3 Mean-Square Stochastic Differential Equations 65310.4 Ergodicity [10-3] 65810.5 Karhunen–Loeve Expansion [10-5] 66510.6 Representation of Bandlimited and Periodic Processes 671

Bandlimited Processes 671Bandpass Random Processes 674WSS Periodic Processes 677Fourier Series for WSS Processes 680

Summary 682Appendix: Integral Equations 682

Existence Theorem 683Problems 686References 699

11 Applications to Statistical Signal Processing 70011.1 Estimation of Random Variables and Vectors 700

More on the Conditional Mean 706Orthogonality and Linear Estimation 708Some Properties of the Operator E 716

11.2 Innovation Sequences and Kalman Filtering 718Predicting Gaussian Random Sequences 722Kalman Predictor and Filter 724Error-Covariance Equations 729

11.3 Wiener Filters for Random Sequences 733Unrealizable Case (Smoothing) 734Causal Wiener Filter 736


“A01_STAR1236_04_SE_FM” — 2011/7/8 — 19:34 — page ix

Contents ix

11.4 Expectation-Maximization Algorithm 738Log-likelihood for the Linear Transformation 740Summary of the E-M algorithm 742E-M Algorithm for Exponential Probability

Functions 743Application to Emission Tomography 744Log-likelihood Function of Complete Data 746E-step 747M-step 748

11.5 Hidden Markov Models (HMM) 749Specification of an HMM 751Application to Speech Processing 753Efficient Computation of P [E|M ] with a Recursive

Algorithm 754Viterbi Algorithm and the Most Likely State Sequence

for the Observations 75611.6 Spectral Estimation 759

The Periodogram 760Bartlett’s Procedure---Averaging Periodograms 762Parametric Spectral Estimate 767Maximum Entropy Spectral Density 769

11.7 Simulated Annealing 772Gibbs Sampler 773Noncausal Gauss–Markov Models 774Compound Markov Models 778Gibbs Line Sequence 779

Summary 783Problems 783References 788

Appendix A Review of Relevant Mathematics A-1A.1 Basic Mathematics A-1

Sequences A-1Convergence A-2Summations A-3Z-Transform A-3

A.2 Continuous Mathematics A-4Definite and Indefinite Integrals A-5Differentiation of Integrals A-6Integration by Parts A-7Completing the Square A-7Double Integration A-8Functions A-8


“A01_STAR1236_04_SE_FM” — 2011/7/8 — 19:34 — page x

x Contents

A.3 Residue Method for Inverse Fourier Transformation A-10Fact A-11Inverse Fourier Transform for psd of Random Sequence A-13

A.4 Mathematical Induction A-17References A-17

Appendix B Gamma and Delta Functions B-1B.1 Gamma Function B-1B.2 Incomplete Gamma Function B-2B.3 Dirac Delta Function B-2References B-5

Appendix C Functional Transformations and Jacobians C-1C.1 Introduction C-1C.2 Jacobians for n = 2 C-2C.3 Jacobian for General n C-4

Appendix D Measure and Probability D-1D.1 Introduction and Basic Ideas D-1

Measurable Mappings and Functions D-3D.2 Application of Measure Theory to Probability D-3

Distribution Measure D-4

Appendix E Sampled Analog Waveforms and Discrete-time Signals E-1

Appendix F Independence of Sample Mean and Variance for NormalRandom Variables F-1

Appendix G Tables of Cumulative Distribution Functions: the Normal,Student t, Chi-square, and F G-1

Index I-1


“A01_STAR1236_04_SE_FM” — 2011/7/8 — 19:34 — page xi

Preface

While significant changes have been made in the current edition from its predecessor, theauthors have tried to keep the discussion at the same level of accessibly, that is, less math-ematical than the measure theory approach but more rigorous than formula and recipemanuals.

It has been said that probability is hard to understand, not so much because of itsmathematical underpinnings but because it produces many results that are counter intuitive.Among practically oriented students, Probability has many critics. Foremost among these arethe ones who ask, “What do we need it for?” This criticism is easy to answer because futureengineers and scientists will come to realize that almost every human endeavor involvesmaking decisions in an uncertain or probabilistic environment. This is true for entire fieldssuch as insurance, meteorology, urban planning, pharmaceuticals, and many more. Another,possibly more potent, criticism is, “What good is probability if the answers it furnishes arenot certainties but just inferences and likelihoods?” The answer here is that an immenseamount of good planning and accurate predictions can be done even in the realm of uncer-tainty. Moreover, applied probability—often called statistics—does provide near certainties:witness the enormous success of political polling and prediction.

In previous editions, we have treaded lightly in the area of statistics and more heavilyin the area of random processes and signal processing. In the electronic version of this book,graduate-level signal processing and advanced discussions of random processes are retained,along with new material on statistics. In the hard copy version of the book, we have droppedthe chapters on applications to statistical signal processing and advanced topics in randomprocesses, as well as some introductory material on pattern recognition.

The present edition makes a greater effort to reach students with more expositoryexamples and more detailed discussion. We have minimized the use of phrases such as,

xi©2012 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved.

“A01_STAR1236_04_SE_FM” — 2011/7/8 — 19:34 — page xii

xii Preface

“it is easy to show. . . ”, “it can be shown. . . ”, “it is easy to see. . . ,” and the like. Also,we have tried to furnish examples from real-world issues such as the efficacy of drugs,the likelihood of contagion, and the odds of winning at gambling, as well as from digitalcommunications, networks, and signals.

The other major change is the addition of two chapters on elementary statistics and itsapplications to real-world problems. The first of these deals with parameter estimation andthe second with hypothesis testing. Many activities in engineering involve estimating para-meters, for example, from estimating the strength of a new concrete formula to estimatingthe amount of signal traffic between computers. Likewise many engineering activities involvemaking decisions in random environments, from deciding whether new drugs are effective todeciding the effectiveness of new teaching methods. The origin and applications of standardstatistical tools such as the t-test, the Chi-square test, and the F -test are presented anddiscussed with detailed examples and end-of-chapter problems.

Finally, many self-test multiple-choice exams are now available for students at the bookWeb site. These exams were administered to senior undergraduate and graduate studentsat the Illinois Institute of Technology during the tenure of one of the authors who taughtthere from 1988 to 2006. The Web site also includes an extensive set of small MATLABprograms that illustrate the concepts of probability.

In summary then, readers familiar with the 3rd edition will see the following significantchanges:

• A new chapter on a branch of statistics called parameter estimation with many illus-trative examples;

• A new chapter on a branch of statistics called hypothesis testing with many illustrativeexamples;

• A large number of new homework problems of varying degrees of difficulty to test thestudent’s mastery of the principles of statistics;

• A large number of self-test, multiple-choice, exam questions calibrated to the materialin various chapters available on the Companion Web site.

• Many additional illustrative examples drawn from real-world situations where theprinciples of probability and statistics have useful applications;

• A greater involvement of computers as teaching/learning aids such as (i) graphicaldisplays of probabilistic phenomena; (ii) MATLAB programs to illustrate probabilisticconcepts; (iii) homework problems requiring the use of MATLAB/ Excel to realizeprobability and statistical theory;

• Numerous revised discussions—based on student feedback—meant to facilitate theunderstanding of difficult concepts.

Henry Stark, IITProfessor Emeritus

John W. Woods, RensselaerProfessor


probability, statistics, and random processes for engineers · “a01_star1236_04_se_fm” —...

Documents