Probability Theory and Random Processes

Communication Systems, 5ed., S. Haykin and M. Moher, John Wiley & Sons, Inc., 2006.

Probability

• Probability theory is based on the phenomena that can be modeled by an experiment with an outcome that is subject to chance.

• Definition: A random experiment is repeated n time (n trials) and the event A is observed m times (m occurrences). The probability is the relative frequency of occurrence m/n.

Probability Based on Set Theory

• Definition: An experiment has K possible outcomes where each outcome is represented as the kth sample sk. The set of all outcomes forms the sample space S. The probability measure P satisfies the

• Axioms:– 0 ≤ P[A] ≤ 1– P[S] = 1– If A and B are two mutually exclusive events (the two events cannot

occur in the same experiment), P[AUB]=P [A] + P[B], otherwise P[AUB] = P[A] + P[B] – P[A∩B]

– The complement is P[Ā] = 1 – P[A]– If A1, A2,…, Am are mutually exclusive events, then P[A1] + P[A2] + … +

P[Am] = 1

Venn Diagrams

A B

Events A and B that are mutually exclusive events in the sample space S.

S

sk Sample can only come from A, B, or neither.

A B

S

sk

Sample can only come from both A and B.

Events A and B are not mutually exclusive events in the sample space S.

Conditional Probability

• Definition: An experiment involves a pair of events A and B where the probability of one is conditioned on the occurrence of the other. Example: P[A|B] is the probability of event A given the occurrence of event B

• In terms of the sets and subsets– P[A|B] = P[A∩B] / P[A]– P[A∩B] = P[A|B]P[B] = P[B|A]P[A]

• Definition: If events A and B are independent, then the conditional probability is simply the elementary probability, e.g. P[A|B] = P[A], P[B|A] = P[B].

Random Variables

• Definition: A random variable is the assignment of a variable to represent a random experiment. X(s) denotes a numerical value for the event s.

• When the sample space is a number line, x = s.• Definition: The cumulative distribution function (cdf)

assigns a probability value for the occurrence of x within a specified range such that FX(x) = P[X ≤ x].

• Properties:– 0 ≤ FX(x) ≤ 1

– FX(x1) ≤ FX(x2), if x1 ≤ x2

Random Variables

• Definition: The probability density function (pdf) is an alternative description of the probability of the random variable X: fX(x) = d/dx FX(x)

• P[x1 ≤ X ≤ x2] = P[X ≤ x2] - P[X ≤ x1]

= FX(x2) - FX(x1)

= fX(x)dx over the interval [x1,x2]

Example Distributions

• Uniform distribution

bx

bxaab

ax

xf X

,0

,1

,0

)(

bx

bxaab

axax

xFX

,1

,

,0

)(

Several Random Variables

• CDF:

• Marginal cdf:

• PDF:

• Marginal pdf:

• Conditional pdf:

yYxXyxF YX ,),(, P

),(),( ,

2

, yxFyx

yxf YXYX

1 ),(,

dvduvuf YX

x

YXX dvduvufxF ),()( ,

duyufyf YXY ),()( ,

dvvxfxf YXX ),()( ,

y

YXY dvduvufyF ),()( ,

)(

),()|( ,

xf

yxfxyf

X

YXY

Statistical Averages

• Expected value:

• Function of a random variable:

• Text Example 5.4

dxxxfX XX E

)(XgY

dxxfXgXgdyyyfY XY EE

Statistical Averages

• nth moments:

• Central moments:

dxxfxX Xnn E

dxxfxX X 22E

dxxfxX Xn

Xn

X E

222 XXXX dxxfxX

E

Mean-square value of X

Variance of X

Joint Moments

• Correlation:

• Covariance:

• Correlation coefficient:

dydxyxfyxYX YXkiki ,, ,E Expected value of the product

- Also seen as a weighted inner product

YXXY

YYXXXY

E

EEEcovCorrelation of the central moment

correlatedstrongly ,1

eduncorrelat,0cov

YX

XY

Random Processes

• Definition: a random process is described as a time-varying random variable

• Mean of the random process:

• Definition: a random process is first-order stationary if its pdf is constant

• Definition: the autocorrelation is the expected value of the product of two random variables at different times

tX

dxxxftXt tXX E

xfxf tXtX 21 XX t Constant mean, variance 22

XX t

2121, tXtXttRX E 2121, ttRttR XX Stationary to second order

Random Processes

• Definition: the autocorrelation is the expected value of the product of two random variables at different times

• Definition: the autocovariance of a stationary random process is

2121, tXtXttRX E

2121, ttRttR XX Stationary to second order

2

21

2121,

XX

XXX

ttR

tXtXttC

E

Properties of Autocorrelation

• Definition: autocorrelation of a stationary process only depends on the time differences

• Mean-square value:

• Autocorrelation is an even function:

• Autocorrelation has maximum at zero:

tXtXRX E

tXRX20 E

XX RR

0XX RR

Example

• Sinusoidal signal with random phase

• Autocorrelation

otherwise,0

-,2

1

,2cos

tf

tfAtX c

c

X

fA

XtXR

2cos2

2

E

As X(t) is compared to itself at another time, we see there is a periodic behavior it in correlation

Cross-correlation

• Two random processes have the cross-correlation

• Wide-sense stationary cross-correlation

uYtXutR

tYtX

XY E,

,

XYXY

YYXX

RuR

uRuRtRtR

,

,,,

Example

• Output of an LTI system when the input is a RP• Text 5.7

Power Spectral Density

• Definition: Fourier transform of autocorrelation function is called power spectral density

• Consider the units of X(t) Volts or Amperes• Autocorrelation is the projection of X(t) onto itself• Resulting units of Watts (normalized to 1 Ohm)

dfefSτR

dτeRfS

fjXX

fjXX

2

2

Properties of PSD

• Zero-frequency of PSD

• Mean-square value

• PSD is non-negative

• PSD of a real-valued RP

dτRS XX 0

dffStX X2E

Which theorem does this property resemble?

0fSX

fSfS XX

Example

• Text Example 5.12– Mixing of a random process with a sinusoidal process

– Autocorrelation

– PSD

tftXtY c2cos

Wide-sense stationary RP (to make it easier)

Uniformly distributed, but not time-varying

cXY fRtYtYR 2cos2

1E

cYcYY ffSffSfS 4

1

PSD of LTI System

• Start with what you know and work the math

dddetXtXhh

dedtXhdtXh

detXthtXth

detYtYfS

dτeRfS

tXthtY

fj

fj

fj

fjY

fjYY

**

*

212

2121

2222111

2

2

2

E

E

E

E

PSD of LTI System

• The PSD reduces to

fSfHfS

dddeRhhfS

dddeRhhfS

XY

fjXY

fjXY

2

0212

01020

210021

212

2121

variablesof Change

210

System shapes power spectrum of input as expected from a filtering like operation

Gaussian Process

• The Gaussian probability density function for a single variable is

• When the distribution has zero mean and unit variance

• The random variable Y is said to be normally distributed as N(0,1)

2

2

2exp

2

1

Y

Y

Y

Y

yyf

2exp

2

1 2yyfY

Properties of a Gaussian Process

• The output of a LTI is Gaussian if the input is Gaussian

• The joint pdf is completely determined by the set of means and autocovariance functions of the samples of the Gaussian process

• If a Gaussian process is wide-sense stationary, then the output of the LTI system is strictly stationary

• A Gaussian process that has uncorrelated samples is statistically independent

Noise

• Shot noise

• Thermal noise

• White noise

• Narrow