# Probability and Random Processes - ?· Probability and Random Processes Main aims I Understand fundamental…

Post on 30-Aug-2018

212 views

Category:

## Documents

0 download

Embed Size (px)

TRANSCRIPT

• Probability and Random Processes

Probability and Random Processes

Jinho ChoiGIST

February 2017

1 / 99

• Probability and Random Processes

What Albert Einstein said:

I As I have said so many times, God doesnt play dice with theworld.

I Two things are infinite: the universe and human stupidity.

So, he does believe in infinity,but not in randomness.

2 / 99

• Probability and Random Processes

Main aims

I Understand fundamental issues of probability theory (pdf,mean, and variance)

I Understand joint and conditional pdf, independence, andcorrelation

I Learn properties of Gaussian random variables and be able toderive Chernoff bound

I Understand random processes with key definitions

I Be able to compute the mean and variance of samples fromrandom processes

3 / 99

• Probability and Random Processes

Probability Theory

Probability Theory

The three axioms with a sample space, , a family of sets, F , forallowable events, and measure Pr():

I The first axiom: The probability of an event, E, is anon-negative real number:

Pr(E) 0.

I The second axiom:Pr() = 1.

I The third axiom: For any countable sequence of mutuallyexclusive events E1, E2, . . .,

Pr(E1 E2 ) =

i=1

Pr(Ei).

4 / 99

• Probability and Random Processes

Probability Theory

Random variables: A random variable is a mapping from thesample space to the set of real numbers.

Sample space (abstract space) Real number

The main idea of random variables is to describe some randomevents by numbers.

real numbers

Event space

event X( )

X: random variable:

X() (,)

5 / 99

• Probability and Random Processes

Probability Theory

Example of random variables: A game of dice

I Before you draw a dice, the number of dots is unknown. This number can be considered as a random variable.I Once a dice is drawn, we have a particular number, which

would be one of {1, . . . , 6}. This is called a realization.

realizations of {2, 3, 4}6 / 99

• Probability and Random Processes

Probability Theory

Cumulative distribution function (CDF):

FX(x) = Pr(X x) ,

where X is the random variable (r.v.) and x is a real number. Bydefinition, the CDF is a nondecreasing function.Example: a dice

Cummulative distribution function (CDF):

FX(x) = Pr(X x),where X is the random variable (r.v.) and x is a real number.

Probability density function (PDF):

fX(x) =d

dxFX(x)

Example: a dice

1 4 5 62 3

2/6

3/6

4/6

5/6

1

1/6

FX(x) = Pr(X x)

x

3

7 / 99

• Probability and Random Processes

Probability Theory

There are different types of r.v.s such as:

I continuous r.v.: X has a continuous value

I discrete r.v.: X has a discrete value

Examples:

I the phase of a sinusoid, : sin(ct+ ) continuous r.v.

I the number of dots of a dice discrete r.v.

8 / 99

• Probability and Random Processes

Probability Theory

A discrete r.v. with binomial distribution

I Consider a random experiement that has two possibleoutcomes. For example, the outcome of this experiment canbe expressed (1 to denote success; 0 to denote failure) as

Y =

{1, with probability p;0, with probability 1 p.

I Consider a sum of n outcomes from independent experiments:

X =

n

j=1

Yj = {0, 1 . . . , n}.

I Then, X is the binomial random variable with parameters nand p, X B(n, p):

Pr(X = j) =

(n

j

)pj(1 p)nj , j = 0, . . . , n.

9 / 99

• Probability and Random Processes

Probability Theory

Continuous r.v., X, has the probability density function (pdf) as

fX(x) =d

dxFX(x) .

I As FX(x) is nondecreasing, fX(x) 0.

I In general,

t

fX(x)dx = FX(t) .

I Since limx FX(x) = 1, fX(x)dx = 1.

For a discrete r.v., the pdf becomes the probability massfunction (pmf) which is actually probability.

I Example of dice:

fX(X = k) Pr(X = k) =1

6, k = 1, 2, . . . , 6.

10 / 99

• Probability and Random Processes

Probability Theory

Mean and VarianceFor a r.v. X, the mean of X (or g(X), where g(x) is a function ofx) is given by

I a continuous r.v.:

E[X] =xfX(x)dx or E[g(X)] =

g(x)fX(x)dx

I a discrete r.v.:

E[X] =

k

xk Pr(xk) or E[g(X)] =

k

g(xk) Pr(xk)

The variance is given by

V ar(X) = E[(X E[X])2]

The mean and variance are used to characterize a random variable.11 / 99

• Probability and Random Processes

Probability Theory

Mean of X B(n, p)

E[X] =n

j=0

j

(n

j

)pj(1 p)nj

=

n

j=1

jn!

j!(n j)!pj(1 p)nj

=

n

j=1

n(n 1)!(j 1)!(n j)!

ppj1(1 p)nj

= np

n

j=1

(n 1)!(j 1)!(n j)!

pj1(1 p)nj

= np(p+ (1 p))n1 = np.

12 / 99

• Probability and Random Processes

Probability Theory

Geometric random variable:

I pmf: Pr(X = k) = (1 p)k1p, k 1Note that

k=1

(1 p)k1p = p

k=0

(1 p)k = p 11 (1 p)

= 1

I Example: The number of independent flips of a coin untilhead first apprears.

I Mean: Letting q = 1 p, it can be shown that

E[X] =

k=1

k(1 p)k1p = p ddq

k=0

qk

= pd

dq

1

1 q= p

1

(1 q)2=

1

p

13 / 99

• Probability and Random Processes

Probability Theory

A continuous r.v. with uniform distributionLet us consider a uniform r.v. X that has the pdf as

fX(x) =

{1A , 0 x A;0, otherwise

The mean is

E[X] = A

0x

1

Adx =

1

A

x2

2

A

x=0

=A

2.

The variance is

E[(X E[X])2] = A

0

(x A

2

)2 1Adx

=

A/2

A/2z2

1

Adz

=1

A 1

3z3A/2

A/2=A2

1214 / 99

• Probability and Random Processes

Probability Theory

More examples on expectationQ) Let X be a r.v. and a and c are constants. Show that

E[aX] = aE[X] and E[X + c] = E[X] + c

A)

E[aX] =

(a x)fX(x)dx = a

xfX(x)dx = aE[X]

E[X + c] =

(x+ c)fX(x)dx =

xfX(x)dx+

cfX(x)dx = E[X] + c

Q) Show that E[X2] = V ar(X) + (E[X])2A)

E[(X E[X])2] = E[X2 2(E[X])X + (E[X])2]= E[X2] 2(E[X])E[X] + (E[X])2= E[X2] (E[X])2

15 / 99

• Probability and Random Processes

Probability Theory

Q) Show that V ar(aX + c) = a2V ar(X)

Q) Suppose that X is a r.v. with mean 1 and variance 3. FindE[3X2 + 2X].A)

E[3X2 + 2X] = 3E[X2] + 2E[X]= 3(V ar(X) + E2[X]) + 2E[X]

= 3 (3 + 12) + 2 1 = 14.

16 / 99

• Probability and Random Processes

Probability Theory

Jensens inequality:For a convex function, g(x),

E[g(X)] g(E[X]) .

A convex function satisfiesg(x1) + (1 )g(x2) g(x1 + (1 )x2), [0, 1]

17 / 99

• Probability and Random Processes

Probability Theory

Gaussian or normal random variable:

fX(x) = N (, 2) =12

exp

((x )

2

22

),

where the mean

E[X] =

xfX(x)dx =

and the variance

E[(X E[X])2] =

(x )2fX(x)dx = 2

18 / 99

• Probability and Random Processes

Probability Theory

Normal or Gaussian pdfs:

19 / 99

• Probability and Random Processes

Probability Theory

Normal or Gaussian cdfs:

20 / 99

• Probability and Random Processes

Probability Theory

Q-function: a tail of the normal pdf.

Pr(X x) = Q(x) 4=

x

12

exp

( t

2

2

)dt

21 / 99

• Probability and Random Processes

Probability Theory

Conditional probabilityThe conditional probability of an event A given B is denoted andgiven by

Pr(A |B) = Pr(A,B)Pr(B)

Q) Find the probability that the face of one dot occurs assumingodd dots are observed:A) We have

Pr(odd) =1

2and

Pr(1, odd) = Pr(1) =1

6.

Hence, it follows

Pr(1| odd) = Pr(1, odd)Pr(odd)

=1

3.

22 / 99

• Probability and Random Processes

Probability Theory

Multiple random variables: The joint pdf is written as

fXY (x, y) =2

xyFXY (x, y) =

2

xyPr(X x, Y y)

Conditional pdf:

fX|Y (x|y) =

{fXY (x,y)fY (y)

, if fY (y) 6= 0.0, otherwise.

Marginalization:

fX(x) =

fXY (x, y)dy or fY (y) =

fXY (x, y)dx

23 / 99

• Probability and Random Processes

Probability Theory

Expectation with two r.v.s:For continuous r.v.s, double-integral should take place:

E[g(X,Y )] =

g(x, y)fXY (x, y)dxdy (continuous)

E[g(X,Y )] =

x

y

g(x, y) Pr(X = x, Y = y) (discrete).

The conditional expectation is defined by

E[X|Y ] =xfX|Y (x|Y )dx (continuous)

E[X|Y ] =

x

xPr(X = x |Y ) (discrete).

Note that E[X|Y ] = g(Y ) is a function of Y , which is a randomvariable.

24 / 99

• Probability and Random Processes

Probability Theory

Q) Show thatE[XY ] = E[Y E[X|Y ]]

A)

E[XY ] =

xyfXY (x, y)dxdy

=

xyfX|Y (x|y)fY (y)dxdy

=

y

(xfX|Y (x|y)dx

)fY (y)dy

=

yE[X|y]fY (y)dy = E[Y E[X|Y ]].

25 / 99

• Probability and Random Processes

Probability Theory

Exponential distribution and memorylessness

I The exponential distribution is given by

f(x;) =

{ex, if x 0;0, o.w.

I The mean and variance are 1/ and 1/2, respectively.

I An exponentially distributed random variable T obeys thefollowing relation:

Pr(T > s+ t |T > s) = Pr(T > t), s, t 0.

This property is called the memorylessness.

26 / 99

• Probability and Random Processes

Probability Theory

Independence and correlationThe joint cdf (more than 2 r.v.s) is written as

FX1,X2, ,Xn(x1, x2, , xn) = Pr(X1 x1, , Xn xn)The joint pdf is given by

fX1,X2, ,Xn(x1, x2, , xn) =n

x1 xnFX1,X2, ,Xn(x1, x2, , xn).

The marginal pdf is given by

fX1(x1) =

x2

xn

fX1,X2, ,Xn(x1, x2, , xn)dx2 dxn.

If X1, , Xn are independent, then

FX1,X2, ,Xn(x1, x2, , xn) = FX1(x1) FXn(xn)

and

fX1,X2, ,Xn(x1, x2, , xn) = fX1(x1) fXn(xn) .27 / 99

• Probability and Random Processes

Probability Theory

I Note: If X and Y are independent,

fX|Y (x|y) =fXY (x, y)

fY (y)= fX(x).

I The correlation of X and Y is defined as

corr(X,Y ) = E[XY ]

and the covariance is defined as

cov(X,Y ) = E[(X E[X])(Y E[Y ])].

If cov(X,Y ) = 0, X and Y are said to be uncorrelated (notuncovarianced).

28 / 99

• Probability and Random Processes

Probability Theory

Joint Gaussian random variables: Let us define the randomvector as

x = [X1 X2 Xn]T.

The random variables, Xi, are jointly Gaussian if the pdf can bewritten as

f(x1, x2, , xn) =1

2 det(C)exp

(1

2(xm)TC1(xm)

),

where the mean vector is m = E[x] = [E[X1] . . . E[Xn]]T and thecovariance matrix is

C = E[(Xm)(Xm)T]

=

E[X1X1] E[X1]E[X1] . . . E[X1Xn] E[X1]E[Xn]... . . . ...E[XnX1] E[Xn]E[X1] . . . E[XnXn] E[Xn]E[Xn]

.29 / 99

• Probability and Random Processes

Probability Theory

30 / 99

• Probability and Random Processes

Probability Theory

Moment Generating Function (MGF)

I Finding mn = E[Xn] (the nth moment)I Define the MGF as

MX(t) = E[etX ].

I Using the Taylor series,

MX(t) = E

[

k=0

1

k!(tX)k

]=

k=0

tkmkk!

I From this, the kth moment can be found as

mk =dk

dtkMX(t)

t=0

.

31 / 99

• Probability and Random Processes

Probability Theory

I The MGF can be seen as the Laplace transform of the pdf,fX(t):

MX(t) =

etXfX(x)dx

Thus, using the inverse Laplace transform, the pdf can befound from the MGF.

I Let X and Y be independent random variables. The MGF ofZ = X + Y is given by

MZ(t) = E[etZ ] = E[et(X+Y )] = MX(t)MY (t).

Thus, the pdf of Z is the inverse Laplace transform ofMX(t)MY (t).

I Thus, the pdf of Z is the convolution of the pdfs of X and Y :fZ(x) = fX(x) fY (x).

32 / 99

• Probability and Random Processes

Probability Theory

Sum of independent Gaussian random variables: Let Xi beindependent Gaussian r.v.s, Xi N (i, 2i ). Then,

Y =

i

Xi

is also a Gaussian r.v.

I The MGF of Xi is Mi(t) = exp(it+12

2i t

2).

I The MGF of Y is

MY (t) =

i

Mi(t) = exp

(

i

it+1

22i t

2

)

Y N

(

i

i,

i

2i

).

33 / 99

• Probability and Random Processes

Probability Theory

Gaussian related distributions

I 2 distribution: The pdf of the 2 distribution with N degreesof freedom is written as

fX(x) =1

(N/2)2N/2x(N/2)1ex/2 x 0,

where (x) =

0 tx1etdt is the Gamma function. If x = n

is an integer,(n) = (n 1)!.

Let X1, X2, , XN are identically independentlly distributed(i.i.d.) random variables (r.v.s). In addition, letXi N (0, 1). Then, Y =

Ni=1X

2i is the

2 r.v. with Ndegrees of freedom.

34 / 99

• Probability and Random Processes

Probability Theory

2 pdf

35 / 99

• Probability and Random Processes

Probability Theory

I The 2 distribution with 2 degrees of freedom is anexponential distribution:

fX(x) =1

2e

x2 , x 0.

I The exponential distribution with parameter (denoted byX Exp()) is given by

f(x;) =

{ex, x 0,0, x < 0.

I The mean and variance of X Exp() are

E[X] =1

V ar(X) =1

2.

36 / 99

• Probability and Random Processes

Probability Theory

I Let X1 and X2 be i.i.d. Gaussian r.v. and Xi N (0, 1).Define Y = X1 + jX2. Then, Y is the circularly symmetriccomplex Gaussian (CSCG) r.v. The mean and variance are

E[Y ] = 0,E[Y 2] = 0, and E[|Y |2] = 2.

I From a zero-mean CSCG random vector to a real-valuedGaussian random vector:

y = x1+jx2 [x1x2

] N

([00

],

[E[x1xT1 ] E[x1xT2 ]E[x2xT1 ] E[x2xT2 ]

])

I Then, what is the distribution of z = Ay if y is a zero-meanCSCG random vector to a real-valued Gaussian?

37 / 99

• Probability and Random Processes

Probability Theory

I The real-valued A:

A A =[

• Probability and Random Processes

Probability Theory

Remarks:

I Let X be a CSCG r.v. Then, |X|2 becomes a 2 r.v.I The Gamma function (n):

(n) =

0tn1etdt, n > 0.

Some properties:(n+ 1) = n(n)

and(n+ 1) = n!

In addition,

(1

2

)=.

39 / 99

• Probability and Random Processes

Probability Theory

Rician and Rayleigh random variables

I The Rayleigh r.v. has a single parameter pdf given by

fX(x) =x

2ex

2/22 , x 0

and the cdf is

FX(x) = 1 ex2/22 , x 0.

If X1 and X2 are independent Gaussian random vairables withmean 0 and variance 2, then X =

X21 +X

22 is the

Rayleigh r.v.

40 / 99

• Probability and Random Processes

Probability Theory

I The Rician pdf is written as

fX(x) =x

2I0

(x2

)e(x

2+2)/22 , x 0,

where I0(x) is the modified Bessel function of zero order. Itcan be shown that if X1 and X2 are independent Gaussian,and X1 N (, 2) and X2 N (0, 2), thenX =

X21 +X

22 is the Rician r.v.

41 / 99

• Probability and Random Processes

Probability Theory

Approximations from binomial to Poisson, to Gaussian

I A (discrete) Poisson random variable with parameter ,denoted by Pois(), has the following distribution:

Pr(X = k) =ek

k!, k = 0, 1, . . .

The followings are often useful:

I A binomial random variable B(n, p) can be approximatedby a Poisson random variable with = np if n is sufficientlylarge.

I A Poisson random variable with can be approximated by aGaussian random variable if is large.

42 / 99

• Probability and Random Processes

Probability Theory

Limit of X B(n, p): (taking a heuristic approach)

Pr(X = k) =

(n

k

)pk(1 p)nk

nk

k!pk

(1 p)n

(1 p)k(for large n)

(np)k

k!

epn

epk(use (1 x) ex, |x| 1)

Let = pn and taking n, which means that p approaches 0(epk 1). Then,

Pr(X = k) ke

k!,

which is the Poisson distribution with parameter .

43 / 99

• Probability and Random Processes

Probability Theory

Sum of independent Poisson rvs is also Poisson

I The MGF of Pois():

MX(t) = E[etX] =

k=0

etkke

k!

= e

k=0

(et)k

k!= eee

t = exp((et 1)).

I Z = X + Y , where X Pois(X) and Y Pois(Y ), has theMGF:

MZ(t) = MX(t)MY (t) = eX(e

t1)eY (et1) = e(X+Y )(e

t1),

which leads to Z Pois(X + Y ).

44 / 99

• Probability and Random Processes

Probability Theory

Standardized Poisson random variable approaches Gaussian

I Let X Pois().I E[X] = and V ar(X) = (check these!)I The standardized Poisson rv becomes

X E[X]V ar(X)

=X

( Z N (0, 1) as ) .

I The MGF is

E[etX ] = E[e

tX

]et = exp((e

t 1))et

= exp

(

(t

+t2

2!+

t3

3!32

+ . . .

) t

)

= exp

(t

2

)as ,

which is the MGF of Z N (0, 1).45 / 99

• Probability and Random Processes

Probability Theory

Transformation Method

I Suppose that U is uniformly distributed in the interval [0, 1].

I From U , we may want to generate r.v., X, that have cdfFX(x).

I To this end, defineZ = F1X (U).

I Then, since Pr(u < t) = t, t [0, 1],

Pr(Z x) = Pr(F1X (u) x) = Pr(u FX(x)) = FX(x).

I That is, Z becomes a r.v. X of cdf FX(x).

46 / 99

• Probability and...