stochastic lecture 5
DESCRIPTION
Stochastic lecture 5TRANSCRIPT
Copyright © Syed Ali Khayam 2009
EE 801 – Analysis of Stochastic Systems
Multiple Random Variables
Muhammad Usman IlyasSchool of Electrical Engineering & Computer ScienceNational University of Sciences & Technology (NUST)Pakistan
Copyright © Syed Ali Khayam 2009
In this lecture, we will cover: Joint Distributions of Multiple Random Variables Functions of Multiple Random Variables Moments of Multiple Random Variables Jointly Normal Random Variables Sums of Random Variables
What will we cover in this lecture?
Copyright © Syed Ali Khayam 2009
Vector Random VariablesPairs of Random Variables
3
Copyright © Syed Ali Khayam 2009
Definition of a Jointly Distributed Random Variable A vector random variable is a mapping from an outcome s of a
random experiment to a vector
: nXZ S S Ì
domain Range or image of X
4
Copyright © Syed Ali Khayam 2009
Definition of a Jointly Distributed Random Variable
Random Experiment SX
X, Y(x, y)
2,: X YZ S S Ì
Image courtesy of www.buzzle.com/
1
2
3
4
5
6
654321
SY
5
Copyright © Syed Ali Khayam 2009
Joint Distributions The joint cumulative distribution function (cdf) of a pair of rvs X
and Y is defined as:
Joint distributions are also called compound distributions
{ }{ }
, ( , ) Pr
Pr , , ,
X YF a b X a Y b
X a Y b a b
= £ £
= £ £ -¥ < < ¥ -¥ < < ¥
6
Copyright © Syed Ali Khayam 2009
Examples of Joint Distributions: Throwing Two Dice Experiment
x
y
pX,Y(x,y)
(1,1)
(6,6)
7
Copyright © Syed Ali Khayam 2009
Examples of Joint Distributions: A Joint Gaussian Distribution
8
Copyright © Syed Ali Khayam 2009
Joint Distributions
Y
Xb
X≤b
Y≤d
d
9
Copyright © Syed Ali Khayam 2009
Joint Distributions
b
X≤b
Y≤d
d
,Integrating shaded region gives ( , )X YF b d
Y
X
10
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs F1:
F2:
F3: If X and Y are continuous rvs then FX,Y(a,b) is also continuous
( ),0 , 1, ,X YF a b a b£ £ -¥ < < ¥
( ) ( )1 2 1 2 , 1 1 , 2 2 and , ,X Y X Ya a b b F a b F a b£ £ £
11
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs F4: ( )
( )
and
,
or
,
, 1
, 0
a b
X Y
a b
X Y
F a b
F a b
¥
-¥
¾¾¾¾¾
¾¾¾¾¾
a
b
,Integrating shaded region gives ( , )X YF a b
Y
X
12
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
( )and
, , 1a b
X YF a b¥
¾¾¾¾¾
a→∞
bb→∞
Y
X
13
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
( )and
, , 1a b
X YF a b¥
¾¾¾¾¾
a→∞
b
b→∞ Y
X
14
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
( )and
, , 1a b
X YF a b¥
¾¾¾¾¾
a→∞
b→∞Y
X
15
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
F5: ( ) ( ), , a
X Y YF a b F b¥¾¾¾¾
b
These distributions are called marginal distributions
a→∞
( ) ( ), , bX Y XF a b F a¥¾¾¾
Y
X
16
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
( ) { }{ }{ } { }
{ } ( )
,lim , Pr
Pr
Pr Pr
Pr
a X Y
Y
F a b X Y b
X Y b
Y b X X
Y b F b
¥ = £ ¥ £
= -¥ £ £ ¥ -¥ £ £
= -¥ £ £ -¥ £ £ ¥ -¥ £ £ ¥
= -¥ £ £ =
17
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
( ) { } { } ( ),lim , Pr Pra X Y YF a b X Y b Y b F b¥ = £ ¥ £ = £ =
b
a→∞
Y
X
18
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
( ) { } { } ( ),lim , Pr Pra X Y YF a b X Y b Y b F b¥ = £ ¥ £ = £ =
b
a→∞
Y
X
19
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
b
a→∞
( ) { } { } ( ),lim , Pr Pra X Y YF a b X Y b Y b F b¥ = £ ¥ £ = £ =Y
X
20
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
F6:
{ } , , , ,Pr and c ( , ) ( , ) ( , ) ( , )X Y X Y X Y X Ya X b Y d F b d F a d F b c F a c< £ < £ = - - +
21
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
b
X≤b
Y≤d
d
Y
X
22
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
b
X≤b
Y≤d
d
,Integrating shaded region gives ( , )X YF b d
Y
X
23
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
b
X≤b
Y≤d
d
c
a
{ }We want Pr ca X b Y d< £ < £
Y
X
24
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
b
X≤b
Y≤d
d
c
aIntegrating the entire region gives FX,Y(b,d)
Y
X
25
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
b
X≤b
Y≤d
d
c
a
Firs
t sub
tract
F(a
,d)
from
FX
,Y (b
,d)
Y
X
26
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
b
X≤b
Y≤d
d
c
a
Then subtract FX,Y (b,c) from FX,Y (b,d)-FX,Y (a,d)
Y
X
27
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
b
X≤b
Y≤d
d
c
a
Finally add FX,Y (a,c) to FX,Y (b,d)-FX,Y (a,d)-FX,Y (b,c)
{ } , , , ,Pr and c ( , ) ( , ) ( , ) ( , )X Y X Y X Y X Ya X b Y d F b d F a d F b c F a c< £ < £ = - - +
Y
X
28
Copyright © Syed Ali Khayam 2009
Joint Probability Density Function The joint probability density function is defined as:
Alternatively:
2
, ,( , ) ( , )X Y X Yf x y F x yx y¶
=¶ ¶
, ,( , ) ( , )yx
X Y X YF x y f a b dadb-¥ -¥
= ò ò
29
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
b
X≤b
Y≤d
d
c
a
{ } ,Pr c ( , )b d
X Y
a c
a X b Y d f x y dydx< £ < £ = ò ò
Y
X
30
Copyright © Syed Ali Khayam 2009
Joint Probability Density Function The joint probability density function must satisfy the following
property:
Moreover:
, ( , ) 1X Yf x y dxdy¥ ¥
-¥ -¥
=ò ò
,
,
( ) ( , )
( ) ( , )
X X Y
Y X Y
f x f x y dy
f y f x y dx
¥
-¥¥
-¥
=
=
ò
ò
31
Copyright © Syed Ali Khayam 2009
Homework Reading Assignment: Examples 4.1 to 4.9 in the book
Homework Problem 4.3 in the textbook Problem 4.9 in the textbook
32
Copyright © Syed Ali Khayam 2009
Independent Random Variables Two random variables X and Y are independent if
Or:
, ( , ) ( ) ( )X Y X YF x y F x F y=
,
,
( , ) ( ) ( ) for discrete rvs
( , ) ( ) ( ) for continuous rvs
X Y X Y
X Y X Y
p x y p x p y
f x y f x f y
=
=
33
Copyright © Syed Ali Khayam 2009
Conditional Probability for Random Variables Conditional probability is an important tool when analyzing
random variables
In general, we are interested in finding the probability that a random variable Y takes on a certain value (or a range of values) given that another random variable X has already taken some value
Conditional probability can be expressed in terms of the joint and marginal probabilities
34
Copyright © Syed Ali Khayam 2009
Conditional Probability for Random Variables
Random Experiment
What is the probability that the sum of the two dice is odd given that X=2?
X
Y
35
Copyright © Syed Ali Khayam 2009
Conditional Probability for Random Variables We can extend the definition of conditional probability to
random variables
( | ) Pr{( ) | ( )}
Pr{ }
Pr{ }
YF y x Y y X x
Y y X x
X x
= £ =
£ ==
=
36
Copyright © Syed Ali Khayam 2009
Conditional Probability for Random Variables We can extend the definition of conditional probability to
random variables
What’s wrong with this definition?
( | ) Pr{( ) | ( )}
Pr{( ) ( )}
Pr{ }
YF y x Y y X x
Y y X x
X x
= £ =
£ ==
=
37
Copyright © Syed Ali Khayam 2009
Conditional Probability for Random Variables
What’s wrong with this definition?
This definition will only work for discrete rvs because Pr{X=a}=0for continuous rvs
Therefore, we need to generalize this definition to encompass continuous rvs
Pr{( ) ( )}( | )
Pr{ }Y
Y y X xF y x
X x
£ ==
=
38
Copyright © Syed Ali Khayam 2009
Properties of Joint CDFs
b
0
Pr{( ) ( )}( | ) lim
Pr{ }Y h
Y y x X x hF y x
x X x h
£ < £ +=
< £ +
a
hY
X
39
Copyright © Syed Ali Khayam 2009
Conditional Probability for Random Variables We need to generalize the definition of conditional probability
to encompass both discrete and continuous rvs
( )
( )
( )
( )
( )
( )
0
, ,
,
Pr{( ) ( )}( | ) lim
Pr{ }
, ,
,
Y h
y yx h
X Y X Y
xx h
XX
xy
X Y
X
Y y x X x hF y x
x X x h
f x y dx dy hf x y dy
hf xf x dx
f x y dy
f x
+
-¥ -¥+
-¥
£ < £ +=
< £ +
¢ ¢ ¢ ¢ ¢ ¢
=
¢ ¢
¢ ¢
=
ò ò ò
ò
ò
40
Copyright © Syed Ali Khayam 2009
Conditional Probability for Random Variables
We can differentiate the conditional CDF to obtain the conditional pdf
( )
( )
, ,
( | )
y
X Y
YX
f x y dy
F y xf x
-¥
¢ ¢
=ò
41
( )( )
, ,( | ) ( | ) X Y
Y YX
f x ydf y x F y x
dy f x= =
Copyright © Syed Ali Khayam 2009
Conditional Probability for Random Variables In other words, a joint pdf can be written as a product of
conditional and marginal pdfs:
For a discrete rv, the same condition can be stated as:
42
( ) ( ), , ( | )X Y Y Xf x y f y x f x=
( ) { } ( ), , Pr , ( | )X Y Y Xp x y X x Y y p y x p x= = = =
Copyright © Syed Ali Khayam 2009
Conditional Expectation Expectation of a conditional joint distribution is defined as
43
{ }( | ) continuous rvs
|
( | ) discrete rvs
Y
Yy
yf y x dy
E Y x
yp y x
¥
-¥
ìïïïïïï= íïïïïïïî
ò
å
Copyright © Syed Ali Khayam 2009
Useful Properties of Conditional Expectation
P1:
P2: For a function h(Y):
The k‐th moment of Y is given by
44
{ }{ } { }|E E Y x E Y=
{ } { }{ }( ) ( ) |E h Y E E h Y x=
{ } { }{ }|k kE Y E E Y x=
Copyright © Syed Ali Khayam 2009
Homework Reading Assignment
Example 4.15 to 4.26 in the textbook
45
Copyright © Syed Ali Khayam 2009
Vector Random VariablesMultiple Random Variables
46
Copyright © Syed Ali Khayam 2009
Joint Distribution of Multiple rvs Most of the ideas that we have studied so far can be directly
extended to more than two jointly‐distributed random variables
Joint pmf of n discrete random variables is:
Conditional pmfs are obtained as
47
( ) { }1 2, , , 1 2 1 1 2 2, , , Pr , , ,
nX X X n n np x x x X x X x X x= = = =
( )( )( )
1 2
1 2 1
, , , 1 2 11 2 1
, , , 1 2 1
, , , ,| , , ,
, , ,n
n
n
X X X n nX n n
X X X n
p x x x xp x x x x
p x x x-
--
-
=
Copyright © Syed Ali Khayam 2009
Joint Distribution of Multiple rvs Most of the ideas that we have studied so far can be directly
extended to more than two jointly‐distributed random variables
Joint pmf of n discrete random variables is:
Conditional pmfs are obtained as
48
( ) { }1 2, , , 1 2 1 1 2 2, , , Pr , , ,
nX X X n n np x x x X x X x X x= = = =
( )( )( )
1 2
1 2 1
, , , 1 2 11 2 1
, , , 1 2 1
, , , ,| , , ,
, , ,n
n
n
X X X n nX n n
X X X n
p x x x xp x x x x
p x x x-
--
-
=
Copyright © Syed Ali Khayam 2009
Conditinal Distribution of Multiple rvs Conditional pdf of a joint pdf is:
Repeatedly applying this expression gives:
49
( )( )( )
1 2
1, 2 1
, , , 1
1 1
, , 1 1
, ,| , ,
, ,n
n
n
X X X n
X n n
X X X n
f x xf x x x
f x x-
-
-
=
( )
( ) ( ) ( ) ( )1 2
1 2 1
, , , 1
1 1 1 1 2 2 1 1
, ,
| , , | , , |
n
n n
X X X n
X n n X n n X X
f x x
f x x x f x x x f x x f x-- - -=
Copyright © Syed Ali Khayam 2009
Conditinal Distribution of Multiple rvs Conditional pdf of a joint pdf is:
Repeatedly applying this expression gives:
50
( )( )( )
1 2
1, 2 1
, , , 11 1
, , 1 1
, ,| , ,
, ,n
n
n
X X X nX n n
X X X n
f x xf x x x
f x x-
--
=
( )
( ) ( ) ( ) ( )1 2
1 2 1
, , , 1
1 1 1 1 2 2 1 1
, ,
| , , | , , |
n
n n
X X X n
X n n X n n X X
f x x
f x x x f x x x f x x f x-- - -=
Copyright © Syed Ali Khayam 2009
Joint Distribution of Multiple rvs Marginal pmf of one rv is obtained by summing over the images
of all other rvs
51
( ) { }
( )1
1 2
2
1 1 1
, , , 1 2
Pr
, , ,n
n
X
X X X nx x
p x X x
p x x x
= =
=å å
Copyright © Syed Ali Khayam 2009
Joint Distribution of Multiple rvs Marginal pmf of one rv is obtained by summing over the images
of all other rvs
52
( ) { }
( )1
1 2
2
1 1 1
, , , 1 2
Pr
, , ,n
n
X
X X X nx x
p x X x
p x x x
= =
=å å
Copyright © Syed Ali Khayam 2009
Joint Distribution of Multiple rvs Joint CDF of n continuous random variables is:
Joint pdf is then obtained as
53
( )
( )
1 2
1 2
1 2
, , , 1 2
, , , 1 2 1
, , ,
, , ,
n
n
n
X X X n
x x x
X X X n n
F x x x
f x x x dx dx-¥ -¥ -¥
¢ ¢ ¢ ¢ ¢= ò ò ò
( ) ( )1 2 1 2, , , 1 2 , , , 1 2
1
, , , , , ,n n
n
X X X n X X X nn
f x x x F x x xx x
¶=
¶ ¶
Copyright © Syed Ali Khayam 2009
Joint Distribution of Multiple rvs Joint CDF of n continuous random variables is:
Joint pdf is then obtained as
54
( )
( )
1 2
1 2
1 2
, , , 1 2
, , , 1 2 1
, , ,
, , ,
n
n
n
X X X n
x x x
X X X n n
F x x x
f x x x dx dx-¥ -¥ -¥
¢ ¢ ¢ ¢ ¢= ò ò ò
( ) ( )1 2 1 2, , , 1 2 , , , 1 2
1
, , , , , ,n n
n
X X X n X X X nn
f x x x F x x xx x
¶=
¶ ¶
Copyright © Syed Ali Khayam 2009
Joint Distribution of Multiple rvs Joint CDF of n continuous random variables is:
Joint pdf is then obtained as
55
( )
( )
1 2
1 2
1 2
, , , 1 2
, , , 1 2 1
, , ,
, , ,
n
n
n
X X X n
x x x
X X X n n
F x x x
f x x x dx dx-¥ -¥ -¥
¢ ¢ ¢ ¢ ¢= ò ò ò
( ) ( )1 2 1 2, , , 1 2 , , , 1 2
1
, , , , , ,n n
n
X X X n X X X nn
f x x x F x x xx x
¶=
¶ ¶
Copyright © Syed Ali Khayam 2009
Joint Distribution of Multiple rvs Joint CDF of n continuous random variables is:
Joint pdf is then obtained as
56
( )
( )
1 2
1 2
1 2
, , , 1 2
, , , 1 2 1
, , ,
, , ,
n
n
n
X X X n
x x x
X X X n n
F x x x
f x x x dx dx-¥ -¥ -¥
¢ ¢ ¢ ¢ ¢= ò ò ò
( ) ( )1 2 1 2, , , 1 2 , , , 1 2
1
, , , , , ,n n
n
X X X n X X X nn
f x x x F x x xx x
¶=
¶ ¶
Copyright © Syed Ali Khayam 2009
Joint Distribution of Multiple rvs A single marginal pdf can be obtained as:
Also, a marginal pdf for a sub‐vector rv can be obtained as:
57
( ) ( )1 1 21 , , , 1 2 2, , ,
nX X X X n nf x f x x x dx dx¥ ¥
-¥ -¥
¢ ¢ ¢ ¢= ò ò
( ) ( )1 1 1 2, , 1 1 , , , 1 2 1, , , , , ,
n nX X n X X X n n nf x x f x x x x dx-
¥
- -
-¥
¢ ¢= ò
Copyright © Syed Ali Khayam 2009
Joint Distribution of Multiple rvs A single marginal pdf can be obtained as:
Also, a marginal pdf for a sub‐vector rv can be obtained as:
58
( ) ( )1 1 21 , , , 1 2 2, , ,
nX X X X n nf x f x x x dx dx¥ ¥
-¥ -¥
¢ ¢ ¢ ¢= ò ò
( ) ( )1 1 1 2, , 1 1 , , , 1 2 1, , , , , ,
n nX X n X X X n n nf x x f x x x x dx-
¥
- -
-¥
¢ ¢= ò
Copyright © Syed Ali Khayam 2009
Joint Distribution of Multiple rvs A single marginal pdf can be obtained as:
Also, a marginal pdf for a sub‐vector rv can be obtained as:
59
( ) ( )1 1 21 , , , 1 2 2, , ,
nX X X X n nf x f x x x dx dx¥ ¥
-¥ -¥
¢ ¢ ¢ ¢= ò ò
( ) ( )1 1 1 2, , 1 1 , , , 1 2 1, , , , , ,
n nX X n X X X n n nf x x f x x x x dx-
¥
- -
-¥
¢ ¢= ò
Copyright © Syed Ali Khayam 2009
Joint Distribution of Multiple rvs A single marginal pdf can be obtained as:
Also, a marginal pdf for a sub‐vector rv can be obtained as:
60
( ) ( )1 1 21 , , , 1 2 2, , ,
nX X X X n nf x f x x x dx dx¥ ¥
-¥ -¥
¢ ¢ ¢ ¢= ò ò
( ) ( )1 1 1 2, , 1 1 , , , 1 2 1, , , , , ,
n nX X n X X X n n nf x x f x x x x dx-
¥
- -
-¥
¢ ¢= ò
Copyright © Syed Ali Khayam 2009
Summary A vector rv is a mapping from outcomes of a random
experiment to a vector
Joint density and distribution functions of a vector rv are:
Marginal densities can be obtained as:
61
: nXZ S S Ì
2
, ,( , ) ( , ),X Y X Yf x y F x yx y¶
=¶ ¶ , ,( , ) ( , )
yx
X Y X YF x y f a b dadb-¥ -¥
= ò ò
( ) ( ), , aX Y YF a b F b¥¾¾¾¾
( ) ( ), , bX Y XF a b F a¥¾¾¾
Copyright © Syed Ali Khayam 2009
Summary A vector rv is a mapping from outcomes of a random
experiment to a vector
Joint density and distribution functions of a vector rv are:
Marginal densities can be obtained as:
62
: nXZ S S Ì
2
, ,( , ) ( , ),X Y X Yf x y F x yx y¶
=¶ ¶ , ,( , ) ( , )
yx
X Y X YF x y f a b dadb-¥ -¥
= ò ò
( ) ( ), , aX Y YF a b F b¥¾¾¾¾
( ) ( ), , bX Y XF a b F a¥¾¾¾
Copyright © Syed Ali Khayam 2009
Summary A vector rv is a mapping from outcomes of a random
experiment to a vector
Joint density and distribution functions of a vector rv are:
Marginal densities can be obtained as:
63
: nXZ S S Ì
2
, ,( , ) ( , ),X Y X Yf x y F x yx y¶
=¶ ¶ , ,( , ) ( , )
yx
X Y X YF x y f a b dadb-¥ -¥
= ò ò
( ) ( ), , aX Y YF a b F b¥¾¾¾¾
( ) ( ), , bX Y XF a b F a¥¾¾¾
Copyright © Syed Ali Khayam 2009
Summary Conditional density is defined as:
For independent rvs, we have:
64
( ) ( ), , ( | )X Y Y Xf x y f y x f x=
, ( , ) ( ) ( )X Y X YF x y F x F y=
,
,
( , ) ( ) ( ) for discrete rvs
( , ) ( ) ( ) for continuous rvs
X Y X Y
X Y X Y
p x y p x p y
f x y f x f y
=
=
Copyright © Syed Ali Khayam 2009
Summary Conditional density is defined as:
For independent rvs, we have:
65
( ) ( ), , ( | )X Y Y Xf x y f y x f x=
, ( , ) ( ) ( )X Y X YF x y F x F y=
,
,
( , ) ( ) ( ) for discrete rvs
( , ) ( ) ( ) for continuous rvs
X Y X Y
X Y X Y
p x y p x p y
f x y f x f y
=
=
Copyright © Syed Ali Khayam 2009
Homework Reading Assignment
Example 4.28, 4.29, 4.30 in the textbook
66
Copyright © Syed Ali Khayam 2009
Functions of Multiple Random Variables
67
Copyright © Syed Ali Khayam 2009
Functions of Multiple Random Variables If we have a function of two discrete random variables, Z=g(X,
Y), we can write its pmf as:
68
( ) ( ){ } { } { }
( ) ( ) ( )
Pr ( , ) Pr ( , ) | Pr ,
for all
( , ) |
Z
Z Y Xx
p z y z x X x y z x X x X x
x
p z p y z x x p x
= Ç = = = =
= å
Copyright © Syed Ali Khayam 2009
Functions of Multiple Random Variables If we have a function of two random variables, Z=g(X, Y), we can
write it in terms of the joint pdf and pmf as:
69
( ) ( ) ( ) ( ) ( )( , ) | ( , ) |Z Y X X Yf z f y z x x f x dx f x z y y f y dy¥ ¥
-¥ -¥
= =ò ò
Copyright © Syed Ali Khayam 2009
Example: Z=X+Y Let Z = g(X, Y) = X+Y If Z is discrete, we can find its pmf as:
Or:
70
( ) { } { }
{ } { }
( ) ( )
Pr Pr
Pr | Pr
Z
x
Y Xx
p z Z z X Y z
Y z x X x X x
p z x p x
= = = + =
= = - = =
= -
å
å
( ) ( ) ( )Z X Yy
p z p z y p y= -å
Copyright © Syed Ali Khayam 2009
Example: Z=X+Y Let Z = g(X, Y) = X+Y If Z is continuous, we have:
The same pdf can also be written as:
71
( ) ( ) ( )Z Y Xf z f z x f x dx
¥
-¥
= -ò
( ) ( ) ( )Z X Yf z f z y f y dy¥
-¥
= -ò
Copyright © Syed Ali Khayam 2009
Example: Z=X+Y Let Z = g(X, Y) = X+Y
PDF of the sum of two continuous random variables is equal to their convolution
72
( ) ( ) ( )Z Y Xf z f z x f x dx
¥
-¥
= -ò
( ) ( ) ( )Z X Yf z f z y f y dy¥
-¥
= -ò
Convolution integral
Copyright © Syed Ali Khayam 2009
Example2: Z=X+Y Let Z = g(X, Y) = X+Y Let X and Y be the packet processing time taken at two routers
connected in series
X and Y are identically‐distributed exponential random variables with parameters λ
Find the pdf of Z.
73
Router 1(Time taken: X)
Router 2(Time taken: Y)
Copyright © Syed Ali Khayam 2009
Example2: Z=X+Y
Z=X+Y
The pdf of X is:
74
( )0
0 0
x
X
e xf x
x
ll -ìï ³ïï= íï <ïïî
Router 1(Time taken: X)
Router 2(Time taken: Y)
Copyright © Syed Ali Khayam 2009
Example2: Z=X+Y
Z=X+Y
The pdf of Y is:
75
( )0
0 0,
y
Y
e y z xf y
y y z x
ll -ìï £ £ -ïï= íï < > -ïïî
Router 1(Time taken: X)
Router 2(Time taken: Y)
Copyright © Syed Ali Khayam 2009
Example2: Z=X+Y
Z=X+Y
In other words, the pdf of Y is:
76
( )( ) 0,
0
z x
Y
e z x x zf y
x z
ll - -ìï - > <ïï= íï >ïïî
Router 1(Time taken: X)
Router 2(Time taken: Y)
Copyright © Syed Ali Khayam 2009
Example2: Z=X+Y The pdf of Z is:
77
( ) ( ) ( )
( ) 2
0 0
2
Z Y X
z zz x x z x x
z
f z f z x f x dx
e e dx e e e dx
ze
l l l l l
l
l l l
l
¥
-¥
- - - - -
-
= -
= =
=
ò
ò ò
This is a 2-stage Erlang PDF ( )1
( 1)!
r r t
R
t ef t
r
ll - -
=-
Copyright © Syed Ali Khayam 2009
Homework Reading Assignment
Example 4.31‐4.34 in the textbook; special emphasis on Example 4.34
Homework Assignment Problem 4.31 in textbook Problem 4.38 in textbook Problem 4.41 in textbook
78
Copyright © Syed Ali Khayam 2009
Moments of Functions of Multiple Random Variables
79
Copyright © Syed Ali Khayam 2009
Expected Value of Functions of Multiple RVs The expected value of a function of multiple rvs, g(X, Y), is given
as:
80
{ }( ) ( )
( ) ( )
,
,
, , , jointly continuous
, , , discrete
X Y
i n X Y i ni n
g x y f x y dxdy X YE Z
g x y p x y X Y
¥ ¥
-¥ -¥
ìïïïïïï= íïïïïïïî
ò ò
åå
Copyright © Syed Ali Khayam 2009
Joint Moments of Multiple RVs The jk‐th joint moment of two rvs, X and Y, is given as:
By setting j=0, we can obtain moments of Y Similarly, k=0 yields moments of X
The (j=1, k=1) moment, E{XY}, is generally called the correlation of X and Y
If E{XY}=0 => X and Y are orthogonal or uncorrelated
81
{ }( )
( )
,
,
, , jointly continuous
, , discrete
j kX Y
j k
j ki n X Y i n
i n
x y f x y dxdy X YE X Y
x y p x y X Y
¥ ¥
-¥ -¥
ìïïïïïï= íïïïïïïî
ò ò
åå
Copyright © Syed Ali Khayam 2009
Joint Central Moments of Multiple RVs The jk‐th central moment of two rvs, X and Y, is:
By setting j=0 and k=2, gives the variance of Y Similarly, setting j=2 and k=0 gives the variance of X
The (j=1, k=1) central moment, E{(X‐E{X})(Y‐E{Y})}, is generally called the covariance of X and Y
A more convenient representation of COV(X,Y) is:
Independent rvs have zero covariance82
{ }( ) { }( ){ }j kE X E X Y E Y- -
{ } { }( ) { }( ){ },COV X Y E X E X Y E Y= - -
{ } { } { } { },COV X Y E XY E X E Y= -
Copyright © Syed Ali Khayam 2009
Correlation Coefficient The correlation coefficient of X and Y is
It can be shown that
Thus correlation coefficient is a normalized measure that quantifies the amount of dependence between two random variables
83
{ } { } { } { },
,X Y
X Y X Y
COV X Y E XY E X E Yr
s s s s-
= =
,1 1X Yr- £ £
Copyright © Syed Ali Khayam 2009
Correlation Coefficient
Source: Wikipedia – Pearson product‐moment correlation coefficient, http://en.wikipedia.org/wiki/Pearson_product‐moment_correlation_coefficient
Copyright © Syed Ali Khayam 2009
Correlation Coefficient
Source: Wikipedia – Pearson product‐moment correlation coefficient, http://en.wikipedia.org/wiki/Pearson_product‐moment_correlation_coefficient
Copyright © Syed Ali Khayam 2009
Correlation Coefficient
Source: Wikipedia – Pearson product‐moment correlation coefficient, http://en.wikipedia.org/wiki/Pearson_product‐moment_correlation_coefficient
Copyright © Syed Ali Khayam 2009
Correlation Coefficient
Source: Wikipedia – Pearson product‐moment correlation coefficient, http://en.wikipedia.org/wiki/Pearson_product‐moment_correlation_coefficient
Copyright © Syed Ali Khayam 2009
Correlation Coefficient
Source: Wikipedia – Pearson product‐moment correlation coefficient, http://en.wikipedia.org/wiki/Pearson_product‐moment_correlation_coefficient
Copyright © Syed Ali Khayam 2009
Correlation Coefficient
Source: Wikipedia – Pearson product‐moment correlation coefficient, http://en.wikipedia.org/wiki/Pearson_product‐moment_correlation_coefficient
Copyright © Syed Ali Khayam 2009
Correlation Coefficient
Source: Wikipedia – Pearson product‐moment correlation coefficient, http://en.wikipedia.org/wiki/Pearson_product‐moment_correlation_coefficient
Copyright © Syed Ali Khayam 2009
Correlation Coefficient
Source: Wikipedia – Pearson product‐moment correlation coefficient, http://en.wikipedia.org/wiki/Pearson_product‐moment_correlation_coefficient
Copyright © Syed Ali Khayam 2009
Useful Properties of Moments of Multiple RVs P1:
If all X’s are independent:
92
{ } { } { } { }1 2 1 2n nE X X X E X E X E X+ + + = + + +
( ) ( ) ( ){ } ( ){ } ( ){ } ( ){ }1 2 1 2n nE g X g X g X E g X E g X E g X=
Copyright © Syed Ali Khayam 2009
Reading Assignment Section 4.7 in the textbook
Examples 4.39‐4.42
93
Copyright © Syed Ali Khayam 2009
Jointly Normal (Gaussian) Random Variables
94
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables The multi‐variate Gaussian pdf has the following form:
Where
95
11 2 1
2
1 1( , ,..., ) exp22
T
X Nf x x x x m x m
1
2
N
m
21 12 1
221 2 2
21 2
N
N
N N N
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables The multi‐variate Gaussian pdf has the following form:
For the Bivariate case:
96
11 2 1
2
1 1( , ,..., ) exp22
T
X Nf x x x x m x m
1
2
m
21 12
221 2
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables The multi‐variate Gaussian pdf has the following form:
For the Bivariate case:
97
11 2 1
2
1 1( , ,..., ) exp22
T
X Nf x x x x m x m
1
2
m
21 12
221 2
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables The multi‐variate Gaussian pdf has the following form:
For the Bivariate case:
98
11 2 1
2
1 1( , ,..., ) exp22
T
X Nf x x x x m x m
1
2
m
21 12
221 2
1 212
1 2
( , )Cov X X
1 2 12 12 1 2( , )Cov X X
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables The multi‐variate Gaussian pdf has the following form:
For the Bivariate case:
99
11 2 1
2
1 1( , ,..., ) exp22
T
X Nf x x x x m x m
1
2
m
21 12 1 2
221 2 1 2
1 212
1 2
( , )Cov X X
1 2 12 12 1 2( , )Cov X X 12 21
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables The multi‐variate Gaussian pdf has the following form:
For the Bivariate case:
100
11 2 1
2
1 1( , ,..., ) exp22
T
X Nf x x x x m x m
1
2
m
21 1 2
22 1 2
1 212
1 2
( , )Cov X X
1 2 12 12 1 2( , )Cov X X 12 21
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables Two random variables X and Y are jointly normal if their pdf has
the following form
101
11 2 1
2
1 1( , ,..., ) exp22
T
X Nf x x x x m x m
21 12 1 2
221 2 1 2
121 2
2 2 2 2 21 1 221 2 1 22
2 1 2
21 2 1
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables Two random variables X and Y are jointly normal if their pdf has
the following form
102
11 2 1
2
1 1( , ,..., ) exp22
T
X Nf x x x x m x m
21 12 1 2
221 2 1 2
121 2
2 2 2 2 21 1 221 2 1 22
2 1 2
21 2 1
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables Two random variables X and Y are jointly normal if their pdf has
the following form
103
11 2 1
2
1 1( , ,..., ) exp22
T
X Nf x x x x m x m
21 1 21
2
22 1 2
11
11
21 12 1 2
221 2 1 2
121 2
2 2 2 2 21 1 221 2 1 22
2 1 2
21 2 1
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables Two random variables X and Y are jointly normal if their pdf has
the following form
104
( )( )
2 2
,2,
, 2,
1exp 2
2 1,
2 1
X X Y YX Y
X X Y YX Y
X Y
X Y X Y
x m x m y m y m
f x y
rs s s sr
ps s r
ì üé ùï ïæ ö æ öæ ö æ öï ï- - - -ê ú- ÷ ÷ ÷ ÷ç ç ç çï ï÷ ÷ ÷ ÷ç ç ç ç- +ê úí ý÷ ÷ ÷ ÷ç ç ç ç÷ ÷ ÷ ÷ï ï÷ ÷ ÷ ÷ç ç ç çê ú- è ø è øè ø è øï ïê úï ïë ûî þ=-
11 2 1
2
1 1( , ,..., ) exp22
T
X Nf x x x x m x m
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables
105Picture from Prof. Hayder Radha’s lecture notes of ECE863 course
( )( )
2 2
,2,
, 2,
1exp 2
2 1,
2 1
X X Y YX Y
X X Y YX Y
X Y
X Y X Y
x m x m y m y m
f x y
rs s s sr
ps s r
ì üé ùï ïæ ö æ öæ ö æ öï ï- - - -ê ú- ÷ ÷ ÷ ÷ç ç ç çï ï÷ ÷ ÷ ÷ç ç ç ç- +ê úí ý÷ ÷ ÷ ÷ç ç ç ç÷ ÷ ÷ ÷ï ï÷ ÷ ÷ ÷ç ç ç çê ú- è ø è øè ø è øï ïê úï ïë ûî þ=-
X
Y
Bivariate Gaussian PDF
-4 -2 0 2 4-4
-2
0
2
4
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables
106Picture from Prof. Hayder Radha’s lecture notes of ECE863 course
( )( )
2 2
,2,
, 2,
1exp 2
2 1,
2 1
X X Y YX Y
X X Y YX Y
X Y
X Y X Y
x m x m y m y m
f x y
rs s s sr
ps s r
ì üé ùï ïæ ö æ öæ ö æ öï ï- - - -ê ú- ÷ ÷ ÷ ÷ç ç ç çï ï÷ ÷ ÷ ÷ç ç ç ç- +ê úí ý÷ ÷ ÷ ÷ç ç ç ç÷ ÷ ÷ ÷ï ï÷ ÷ ÷ ÷ç ç ç çê ú- è ø è øè ø è øï ïê úï ïë ûî þ=-
X
Y
Bivariate Gaussian PDF
-4 -2 0 2 4-4
-2
0
2
4
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables
107Picture from Prof. Hayder Radha’s lecture notes of ECE863 course
( )( )
2 2
,2,
, 2,
1exp 2
2 1,
2 1
X X Y YX Y
X X Y YX Y
X Y
X Y X Y
x m x m y m y m
f x y
rs s s sr
ps s r
ì üé ùï ïæ ö æ öæ ö æ öï ï- - - -ê ú- ÷ ÷ ÷ ÷ç ç ç çï ï÷ ÷ ÷ ÷ç ç ç ç- +ê úí ý÷ ÷ ÷ ÷ç ç ç ç÷ ÷ ÷ ÷ï ï÷ ÷ ÷ ÷ç ç ç çê ú- è ø è øè ø è øï ïê úï ïë ûî þ=-
X
Y
Bivariate Gaussian PDF
-4 -2 0 2 4-4
-2
0
2
4
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables
108
( )( )
2 2
,2,
, 2,
1exp 2
2 1,
2 1
X X Y YX Y
X X Y YX Y
X Y
X Y X Y
x m x m y m y m
f x y
rs s s sr
ps s r
ì üé ùï ïæ ö æ öæ ö æ öï ï- - - -ê ú- ÷ ÷ ÷ ÷ç ç ç çï ï÷ ÷ ÷ ÷ç ç ç ç- +ê úí ý÷ ÷ ÷ ÷ç ç ç ç÷ ÷ ÷ ÷ï ï÷ ÷ ÷ ÷ç ç ç çê ú- è ø è øè ø è øï ïê úï ïë ûî þ=-
If we set the exponents involving x and y in the above expression to a constant K, we obtain the equation for an ellipse
( )( )2
,
, 2,
1exp
2 1,
2 1
X Y
X Y
X Y X Y
K
f x yr
ps s r
ì üï ïï ï-ï ïí ýï ï-ï ïï ïî þ=-
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables
109
( )( )
2 2
,2,
, 2,
1exp 2
2 1,
2 1
X X Y YX Y
X X Y YX Y
X Y
X Y X Y
x m x m y m y m
f x y
rs s s sr
ps s r
ì üé ùï ïæ ö æ öæ ö æ öï ï- - - -ê ú- ÷ ÷ ÷ ÷ç ç ç çï ï÷ ÷ ÷ ÷ç ç ç ç- +ê úí ý÷ ÷ ÷ ÷ç ç ç ç÷ ÷ ÷ ÷ï ï÷ ÷ ÷ ÷ç ç ç çê ú- è ø è øè ø è øï ïê úï ïë ûî þ=-
If we set the exponents involving x and y in the above expression to a constant K, we obtain the equation for an ellipse
( )( )2
,
, 2,
1exp
2 1,
2 1
X Y
X Y
X Y X Y
K
f x yr
ps s r
ì üï ïï ï-ï ïí ýï ï-ï ïï ïî þ=-
This is the equation for an ellipse
Copyright © Syed Ali Khayam 2009
( )( )2
,
, 2,
1exp
2 1,
2 1
X Y
X Y
X Y X Y
K
f x yr
ps s r
ì üï ïï ï-ï ïí ýï ï-ï ïï ïî þ=-
Jointly Normal Random Variables
110
The orientation of the ellipse depends on the value of correlation ρ
Copyright © Syed Ali Khayam 2009
( )( )2
,
, 2,
1exp
2 1,
2 1
X Y
X Y
X Y X Y
K
f x yr
ps s r
ì üï ïï ï-ï ïí ýï ï-ï ïï ïî þ=-
Jointly Normal Random Variables
111
The orientation of the ellipse depends on the value of correlation ρ
When ρ≠0, we have:
Picture from Prof. Hayder Radha’s lecture notes of ECE863 course
Copyright © Syed Ali Khayam 2009
( )( )2
,
, 2,
1exp
2 1,
2 1
X Y
X Y
X Y X Y
K
f x yr
ps s r
ì üï ïï ï-ï ïí ýï ï-ï ïï ïî þ=-
Jointly Normal Random Variables
112
The orientation of the ellipse depends on the value of correlation ρ
When ρ≠0, we have:
Picture from Prof. Hayder Radha’s lecture notes of ECE863 course
Copyright © Syed Ali Khayam 2009
( )( )2
,
, 2,
1exp
2 1,
2 1
X Y
X Y
X Y X Y
K
f x yr
ps s r
ì üï ïï ï-ï ïí ýï ï-ï ïï ïî þ=-
Jointly Normal Random Variables
113
The orientation of the ellipse depends on the value of correlation ρ
When ρ≠0, we have:
Picture from Prof. Hayder Radha’s lecture notes of ECE863 course
Copyright © Syed Ali Khayam 2009
( )( )2
,
, 2,
1exp
2 1,
2 1
X Y
X Y
X Y X Y
K
f x yr
ps s r
ì üï ïï ï-ï ïí ýï ï-ï ïï ïî þ=-
Jointly Normal Random Variables
114
When ρ=0 (i.e., X & Y are uncorrelated), we have:
The ellipses are parallel to the X and Y axis
Picture from Prof. Hayder Radha’s lecture notes of ECE863 course
Copyright © Syed Ali Khayam 2009
( )( )2
,
, 2,
1exp
2 1,
2 1
X Y
X Y
X Y X Y
K
f x yr
ps s r
ì üï ïï ï-ï ïí ýï ï-ï ïï ïî þ=-
Jointly Normal Random Variables
115
When ρ=0 (i.e., X & Y are uncorrelated), we have:
The ellipses are parallel to the X and Y axis
Picture from Prof. Hayder Radha’s lecture notes of ECE863 course
Copyright © Syed Ali Khayam 2009
( )( )2
,
, 2,
1exp
2 1,
2 1
X Y
X Y
X Y X Y
K
f x yr
ps s r
ì üï ïï ï-ï ïí ýï ï-ï ïï ïî þ=-
Jointly Normal Random Variables
116
When ρ=0 (i.e., X & Y are uncorrelated), we have:
The ellipses are parallel to the X and Y axis
Picture from Prof. Hayder Radha’s lecture notes of ECE863 course
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables
117
As ρ=0 increases towards 1, X and Y become more and more correlated and the ellipses become narrower
Picture from Prof. Hayder Radha’s lecture notes of ECE863 course
X
Y
Bivariate Gaussian PDF
-4 -2 0 2 4-4
-2
0
2
4
Copyright © Syed Ali Khayam 2009
Jointly Normal Random Variables
118
As ρ=0 increases towards 1, X and Y become more and more correlated and the ellipses become narrower
Picture from Prof. Hayder Radha’s lecture notes of ECE863 course
XY
Bivariate Gaussian PDF
-4 -2 0 2 4-4
-2
0
2
4
Copyright © Syed Ali Khayam 2009
n Jointly Normal Random Variables Random variables X1, X2,…Xn are jointly normal if their pdf has
the following form:
where:
119
( ) ( )( ) ( ){ }( )1 2
1
, , , 1 2 1/ 2/2
1exp
2, , ,2n
T
X X X nX n
x m K x mf x f x x x
Kp
--- -
=
{ } { } { }
{ } { } { }
{ } { } { }
1 2
1 2
1 1 2 1
2 1 1 2
1 2
var
var
var
n
n
n
n
n n n
x x x x
m m m m
X COV X X COV X X
COV X X X COV X XK
COV X X COV X X X
é ù= ê úë ûé ù= ê úë ûé ùê úê úê úê ú= ê úê úê úê úê úë û
Covariance matrix
Copyright © Syed Ali Khayam 2009
Reading Assignment Section 4.8 in the textbook
Examples 4.45‐4.48
120