5.1 expected value of a function of random variables 5-ho.pdf · expected value of a function of...

1

Chapter 5. Operations on Multiple Random

Variables

0. Introduction

1. Expected Value of a Function of Random Variables

2. Jointly Gaussian Random Variables

3. Transformations of Multiple Random Variables

4. Linear Transformations of Gaussian Random Variables

5. Sampling and Some Limit Theorems

1

5.1 Expected Value of a Function of Random Variables

,

[ ( , )] ( , ) ( , )X Y

E g X Y g x y f x y dxdy∞ ∞

−∞ −∞

= ∫ ∫

11 1 , , 1 1

[ ( , , )] ( , , ) ( , , )N

N N X X N NE g X X g x x f x x dx dx

∞ ∞

−∞ −∞

= ∫ ∫ ⋯

⋯ ⋯ ⋯ ⋯ ⋯

, ,

[ ( )] ( ) ( , ) ( ) ( , )X Y X Y

E g X g x f x y dxdy g x f x y dydx∞ ∞ ∞ ∞

−∞ −∞ −∞ −∞

= =∫ ∫ ∫ ∫

( ) ( )X

g x f x dx∞

−∞

= ∫

2

2


1

1 1 1

[ ( , , )] [ ] [ ] [ ]N N N

N i i i i i i

i i i

E g X X E X E X E Xα α α

= = =

= = =∑ ∑ ∑⋯

Ex 5.1-1:

1

1

( , , )N

N i i

i

g X X Xα

=

=∑⋯

Joint moments

,

[ ] ( , )n k n k

nk X Ym E X Y x y f x y dxdy

∞ ∞

−∞ −∞

= = ∫ ∫

10[ ]m E X=

01[ ]m E Y= 11

[ ]m E XY=

Correlation of & :X Y 11[ ]

XYR E XY m= =

3


Ex 5.1-2:

& -- uncorrelated [ ] [ ] [ ]X Y E XY E X E Y⇔ =

& -- orthogonal [ ] 0X Y E XY⇔ =

independent uncorrelated⇒

⇐×

,

[ ] ( , ) ( ) ( )X Y X Y

E XY xyf x y dxdy xyf x f y dxdy∞ ∞ ∞ ∞

−∞ −∞ −∞ −∞

= =∫ ∫ ∫ ∫

( ) ( ) [ ] [ ]X Y

xf x dx yf y dy E X E Y∞ ∞

−∞ −∞

= =∫ ∫[ ] 3E X =

2[ ] 11E X =

6 22Y X= − +

[ ] 6 [ ] 22 4E Y E X= − + =

2[ ] [ ( 6 22)] 6 [ ] 22 [ ] 0

XYR E XY E X X E X E X= = − + = − + = orthogonal⇒

4

3


[ ] [ ] [ ] 12XY

R E XY E X E Y= ≠ = NOT uncorrelated⇒

Y aX b= +

2[ ] [ ( )] [ ] [ ]E XY E X aX b aE X bE X= + = +

2[ ] [ ] [ ] [ ] ( [ ]) [ ]E X E Y E X E aX b a E X bE X= + = +

0 [ ] [ ] [ ] uncorrelateda E XY E X E Y= ⇒ = ⇒

2[ ]orthogonal

[ ]

b E X

a E X= − ⇒

3 2 5

3,2,5 1 2 3[ ]m E X X X=

5


Joint central moments

[( ) ( ) ]n k

nkE X X Y Yµ = − −

10[ ] 0E X Xµ = − =

Covariance of & :X Y

01[ ] 0E Y Yµ = − =

2 2

20[( ) ]

XE X Xµ σ= − =

2 2

02[( ) ]

YE Y Yµ σ= − =

11[( )( )]

XYC E X X Y Yµ= = − −

[( )( )] [ ]XY

C E X X Y Y E XY XY YX XY= − − = − − +

[ ] [ ] [ ] [ ] [ ]XY

E XY XE Y YE X XY R E X E Y= − − + = −

uncorrelated 0XY

C⇔ =

6

4


Correlation coefficient of & :X Y

11

20 02

( ) ( )XY

X Y X Y

C X X Y YE

µρ

σ σ σ σµ µ

− −= = =

orthogonal [ ] [ ]XY

C E X E Y⇒ = −

XY X YC ρσ σ=

7


11

20 02

( ) ( )XY

X Y X Y

C X X Y YE

µρ

σ σ σ σµ µ

− −= = =

1 1ρ− ≤ ≤

uncorrelated 0ρ⇔ =

1Y X ρ= ⇒ =

2 1Y X ρ= − ⇒ = −

8

5


Ex 5.1-3:

1

N

i i

i

X Xα

=

=∑

1 1

[ ] [ ] [ ]N N

i i i i

i i

E X E X E Xα α

= =

= =∑ ∑

1

1

N

i

i

α

=

=∑

1

[ ] ( )N

i i i

i

X E X X Xα

=

− = −∑

2 2

1 1

[( ) ] ( ) ( )N N

X i i i j j j

i j

E X X E X X X Xσ α α

= =

= − = − −

∑ ∑

,

1 1 1 1

[( )( )]i j

N N N N

i j i i j j i j X X

i j i j

E X X X X Cα α α α

= = = =

= − − =∑∑ ∑∑

2

,

,

0,

i

i j

X

X X

i jC

i j

σ ==

≠

2 2 2

1

i

N

X i X

i

σ α σ

=

⇒ =∑

's uncorrelatedi

X

9

5.3 Jointly Gaussian Random Variables2 2

2 2 2

1 ( ) 2 ( )( ) ( )

2(1 )

,2

1( , )

2 1

X YX Y

x X x X y Y y Y

X Y

X Y

f x y e

ρ

σ σρ σ σ

πσ σ ρ

− − − − −− +

− =

−

, ,

2

( , ) ( , )

1

2 1

X Y X Y

X Y

f x y f X Y

πσ σ ρ

≤

=

−

[ ]E X X= [ ]E Y Y=

2 2[( ) ]

XE X X σ− =

[( )( )]X Y

E X X Y Y ρσ σ− − =

10

6

5.3 Jointly Gaussian Random Variables2

2

( )

2

,

1( ) ( , )

2

X

x X

X X Y

X

f x f x y dy eσ

πσ

−

−∞

−∞

= = =∫ ⋯

2

2

( )

2

,

1( ) ( , )

2

Y

y Y

Y X Y

Y

f y f x y dx e σ

πσ

−

−∞

−∞

= = =∫ ⋯

,

0 ( , ) ( ) ( )X Y X Yf x y f x f yρ = ⇒ =

jointly gaussian & uncorr. indep.⇒

Ex 5.3-1:

1cos sinY X Yθ θ= +

2sin cosY X Yθ θ= − +

11

5.3 Jointly Gaussian Random Variables

1 2, 1 1 2 2

[( )( )]Y Y

C E Y Y Y Y= − −

[{( )cos ( )sin }{ ( )sin ( )cos }]E X X Y Y X X Y Yθ θ θ θ= − + − − − + −

2 2 2 2( )sin cos [cos sin ]Y X XY

Cσ σ θ θ θ θ= − + −

2 2 2 21 1( )sin 2 cos2 ( )sin 2 cos2

2 2Y X XY Y X X Y

Cσ σ θ θ σ σ θ ρσ σ θ= − + = − +

1 2

1

, 2 2

210 tan

2

X Y

Y Y

X Y

Cρσ σ

θσ σ

−

= ⇒ = −

12

7


1 1

2 2

3 3

x X

x X x X

x X

−

− = − −

11 12 13

21 22 23

31 32 33

X

C C C

C C C C

C C C

=

1

1 2 3

1( ) ( )

2, , 1 2 3 1/ 2/ 2

1( , , )

(2 )

T

Xx X C x X

X X XN

X

f x x x eCπ

−

− − −

=

1 1 21 1 2

1 2 2

1 2 2

22

1

22

2

1

1

11

X X XX X X

X X

X X X

X X X

C C

ρ

σ σ σσ ρσ σ

ρρρσ σ σ

σ σ σ

−

− = ⇒ = −−

1 2

2 2 2(1 )

X X XC σ σ ρ= −

13


1 2 3Properties of jointly gaussian r.v.'s , , & :X X X

1. 1st & 2nd moments p.d.f⇒

2. uncorr. indep.⇒

3. linear transforms of gaussian r.v.'s is also jointly gaussian.

1 2, 1 2

4. marginal density ( , ) is also jointly gaussian.X Xf x x

1 2 31 2 3,

5. conditional density ( , ) is also jointly gaussian.X X Xf x x x

14

8

Recall: Leibniz’s Rule

• Let � � = � � �,� ��

� �

• Then

��

��= � � � ,�

��

��− � � , �

��

��+ �

�� ,�

��

� �

� �

15

5.4 Transformations of Multiple Random Variables

One function 1 2( , )Y g X X=

1 2, 1 2

( , )X Xf x x

1 2

1 2

1 2 , 1 2 1 2

( , )

( ) [ ( , ) ] ( , )Y X X

g x x y

F y P g X X y f x x dx dx≤

= ≤ = ∫∫

( )( ) Y

Y

dF yf y

dy=

Ex 5.4-1: 1 2positive r.v.'s & X X

1

2

XY

X=

2

1 2

1

, 1 2 1 20 0

2

( ) [ ] ( , )yx

Y X X

XF y P y f x x dx dx

X

∞

= ≤ = ∫ ∫ 0y >

16

9


1 22 , 2 2 2

0

( )( ) ( , )Y

Y X X

dF yf y x f yx x dx

dy

∞

= = ∫ 0y >

Use Leibniz’s Rule

17


1 2, 1 2

( , )X Xf x x

2 2

2 2

( ) ( , ) ( , )( ) ( , ) ( , )

y yY

Yy y

dF y I y x I y xf y I y y I y y dx dx

dy y y− −

∂ ∂= = + − + =

∂ ∂∫ ∫

Ex 5.4-2:2 2

1 2Y X X= +

2 2

2

2 2 1 22

2 2

1 2 , 1 2 1 2( ) [ ] ( , )

y y x

Y X Xy y x

F y P X X y f x x dx dx−

− − −

= + ≤ = ∫ ∫ 2 2( , )

y

y

I y x dx−

= ∫2 2

2

2 2 1 22

2 , 1 2 1( , ) ( , )

y x

X Xy x

I y x f x x dx−

− −

= ∫

1 2 1 2

2 2 2 22

, 2 2 , 2 22 2 2 2

2 2

( , )( , ) ( , )

X X X X

I y x y yf y x x f y x x

y y x y x

∂= − + − −

∂ − −

Using Leibniz’s Rule

18

10


2

2

( , )( )

y

Yy

I y xf y dx

y−

∂=

∂∫

1 2 1 2

2 2 2 2

, 2 2 , 2 2 22 2

2

{ ( , ) ( , )}y

X X X Xy

yf y x x f y x x dx

y x−

= − + − −

−∫

HW: Solve Problem 5.4-3.

2 2

1 1 2 2

2 2

1 2

2

2 (1 )

, 1 22 2

1( , )

2 1

X

x x x x

X X

X

f x x e

ρ

σ ρ

πσ ρ

− +−

−

=

−

2 2

1 2Y X X= +

( ) ?Yf y⇒ =

0ρ =

19


Multiple functions 1 1 1 2

1 2

2 2 1 2

( , )( , )

( , )

Y T X XT X X

Y T X X

= =

1 2, 1 2

( , )X Xf x x

1 2 1 2, 1 2 , 1 2( , ) ( , )

Y Y X Xf y y f x x J=

1

1 1 1 21

1 2 1

2 2 1 2

( , )( , )

( , )

x T y yT y y

x T y y

−

−

−

= =

1 1

1 1

1 2

1 1

2 2

1 2

T T

y yJ

T T

y y

− −

− −

∂ ∂

∂ ∂=

∂ ∂

∂ ∂

jacobian

20

11


1 1 1 2 1 2 1

1 2

2 2 1 2 1 2 2

( , )( , )

( , )

Y T X X aX bX Xa bT X X

Y T X X cX dX c d X

+ = = = = +

Ex 5.4-3:

1 2

1 2

1 2 1 2,

, 1 2

( , )

( , )X X

Y Y

dy by cy ayf

ad bc ad bcf y yad bc

− − +

− −=

−

1

1 1 1 2 11

1 2 1

2 22 1 2

( , ) 1( , )

( , )

x T y y yd bT y y

x c a yad bcT y y

−

−

−

− = = = −−

1 1

1 1

1 2

1 1

2 2

1 2

1

T T

y yJ

ad bcT T

y y

− −

− −

∂ ∂

∂ ∂= =

−∂ ∂

∂ ∂

21

5.5 Linear Transformations of Gaussian Random Variables

1 1 11 12

21 222 2

Y X a aY A AX X

a aY X

= = = =

[ ] [ ] [ ]E Y E AX AE X= =

[( )( ) ] [ ( )( ) ]T T T

YC E Y Y Y Y E A X X X X A= − − = − −

[( )( ) ]T T T

XAE X X X X A AC A= − − =

1 1 1T

Y XC A C A

− − − −

⇒ =

1

1 2

1( ) ( )

2, 1 2 1/ 2/ 2

1( , )

(2 )

T

Xx X C x X

X X N

X

f x x eCπ

−

− − −

=

1 2, 1 2( , ) ?

Y Yf y y =

2det( ) det( ) det( )Y X

C A C=

⇒ � = ��

22

12


1 1 11( ) ( )

21/ 2/ 2

1

(2 ) det( )

T

XA y X C A y X

N

X

e

C Aπ

− − −

− − −

=

1 2 1 2

1

, 1 2 ,

1( , ) ( )

det( )Y Y X Xf y y f A y

A

−

=

1 1 1 1 1( ) ( ) ( ) ( )

T T T

X XA y X C A y X y Y A C A y Y

− − − − − −

− − = − −

1( ) ( )

T

Yy Y C y Y

−

= − −

1

1 2

1( ) ( )

2, 1 2 1/ 2/ 2

1( , )

(2 )

T

Yy Y C y Y

Y Y N

Y

f y y eCπ

−

− − −

⇒ =

1/ 2 1/ 2

det( )X Y

C A C=

(gaussian)

23


1

1 2

1( ) ( )

2, 1 2 1/ 2

1( , )

(2 )

T

Yy Y C y Y

Y Y

Y

f y y eCπ

−

− − −

⇒ =

Ex 5.5-1: 1 1 1

2 2 2

1 2

3 4

Y X XA

Y X X

− = =

1

2

[ ] 0

[ ] 0

E X

E X

=

4 3

3 9X

C

=

1 1

2 2

[ ] [ ] 0

[ ] [ ] 0

E Y E XA

E Y E X

= =

1 2 4 3 1 3 28 66

3 4 3 9 2 4 66 252

T

Y XC AC A

− − = = = − −

1 2

1 2

660.786

28 252

Y Y

Y Y

Cρ

σ σ

−

= = = −

24

13

5.7 Sampling and Some Limit Theorems

sampling and estimation

1

1ˆ

N

N n

n

x xN

=

= ∑ -- estimate of average of samplesN

estimator of mean of r.v. X

1

1ˆN

N n

n

X XN

=

= ∑ (sample mean)

r.v.

25


estimator of power of r.v. X�

2 2

1

1N

N n

n

X XN

=

= ∑

estimator of variance of r.v. X�2 2

1

1 ˆ( )

1

N

X n N

n

X XN

σ

=

= −

−

∑

1 1

1 1ˆ[ ] [ ] [ ]

N N

N n n

n n

E X E X E X XN N

= =

= = =∑ ∑(sample mean estimator is unbiased.)

2

2

1 1 1 1

1 1 1ˆ[ ] [ ] [ ]

N N N N

N n n n m

n n n m

E X E X X E X XN N N

= = = =

= =∑ ∑ ∑∑

2[ ] [ ]

n mn m E X X E X= ⇒ =

2[ ]

n mn m E X X X≠ ⇒ =

indep.2 2 2 2 2

2

1 1{ [ ] ( ) } { ( 1) }NE X N N X X N X

N N= + − = + −

26

14


2 2 2 2 2 21 1 1{ [ ] ( 1) } { [ ] }

XE X N X X E X X

N N Nσ= + − − = − =

( )2

2ˆ

2 2

ˆ1 1

NX X

NP X X

N

σσ

ε

ε ε

− < ≥ − = −

(Chebychev's ineq.)

1N→∞

→

ˆN N

X X→∞

→ w.p. 1 (with probability 1)

Ex 5.7-1: 0.05Xε = 50N =

( )ˆ0.95

NP X X ε− < ≥ ⇒

2

21 0.95 160

50(0.05 )

X

XX

X

σ

σ− ≥ ⇒ ≥

2 2 2 2 2 2

ˆ

ˆ ˆ ˆ ˆ[( ) ] [ 2 ] [ ]

N

N N N NX

E X X E X XX X E X Xσ = − = − + = −

27


�2 2

1

1N

N n

n

X XN

=

= ∑�2 2

1

1 ˆ( )1

N

X n N

n

X XN

σ

=

= −

−

∑

�2 2 2 2 2

1 1

1 1[ ] [ ] [ ] [ ]

N N

N n

n n

E X E X E X E X XN N

= =

= = = =∑ ∑ (unbiased)

�2 2 2 2

1 1

1 1ˆ ˆ ˆ[ ] [( ) ] [ 2 ]

1 1

N N

X n N n n N N

n n

E E X X E X X X XN N

σ

= =

= − = − +

− −∑ ∑

2 2

1 1 1 1 1

1 1ˆ[ ] [ ] [ ] ( 1)N N N N N

n N n m n m

n n m n m

E X X E X X E X X X N XN N

= = = = =

= = = + −∑ ∑ ∑ ∑∑

�2 2 2 2 2

2 2 2 22{ ( 1) } { ( 1) }[ ]

1X X

N X X N X X N XE X X

Nσ σ

− + − + + −= = − =

−

(unbiased)

28

15


�2 2 2 2

1

1 1ˆ( ) [(0.1 3.973) (12.0 3.973) ]1 10

N

X n N

n

X XN

σ

=

= − = − + + −

−∑ ⋯

1

1 1ˆ(0.1 0.4 0.9 12.0) 3.973

11

N

N n

n

X X VN

=

= = + + + + =∑ ⋯

Ex 5.7-2:

11 samples (0.1, 0.4, 0.9,1.4, 2.0, 2.8, 3.7, 4.8, 6.4, 9.2,12.0 )V

sample mean

sample variance

214.75V=

[ ] 4E X V=2

16Xσ =

29


ˆlim [ [ ] ] 1, 0

NN

P X E X ε ε→∞

− < = ∀ >

Weak Law of Large Numbers

Strong Law of Large Numbers

ˆ{lim [ ]} 1N

N

P X E X→∞

= =

30

5.1 expected value of a function of random variables 5-ho.pdf · expected value of a function of...

Documents