11

Multivariate Multivariate DistributionsDistributions ch4

22

Multivariable DistributionsMultivariable Distributions It may be favorable to take more than one measurement on

a random experiment.– The data may then be collected in pairs of (xi, yi).

Def.4.1-1: X & Y are two discrete R.V. defined over the support S. The probability that X=x, Y=y is denoted as f(x,y)=P(X=x,Y=y). f(x,y) is the joint probability mass function (joint p.m.f.) of X and Y:– 0≤f(x,y)≤1; ΣΣ(x,y) S∈ f(x,y)=1; P[(X,Y) A]=ΣΣ∈ (x,y) A∈ f(x,y), A S.⊆

33

Illustration ExampleIllustration Example Ex.4.1-3: Roll a pair of dice: X is the smaller and Y is the larger.

– The outcome is (3, 2) or (2, 3) X=2 & Y=3 with 2/36 probability. ⇒– The outcome is (2, 2) X=2 & Y=2 with 1/36 probability. ⇒– Thus, the joint p.m.f. of X and Y is

11/36 6 2/36 2/36 2/36 2/36 2/36 1/36

9/36 5 2/36 2/36 2/36 2/36 1/36

7/36 4 2/36 2/36 2/36 1/36

5/36 3 2/36 2/36 1/36

3/36 2 2/36 1/36

1/36 1 1/36

1 2 3 4 5 6

11/36 9/36 7/36 5/36 3/36 1/36Marginal p.m.f.

y

x

44

Marginal Probability and IndependenceMarginal Probability and Independence Def.4.1-2: X and Y have the joint p.m.f f(x,y) with space S.

– The marginal p.m.f. of X is f1(x)=Σyf(x,y)=P(X=x), x S∈ 1.

– The marginal p.m.f. of Y is f2(y)=Σxf(x,y)=P(Y=y), y S∈ 2. X and Y are independent iff P(X=x, Y=y)=P(X=x)P(Y=y), namely,

f(x,y)=f1(x)f2(y), x S∈ 1, y S∈ 2.

– Otherwise, X and Y are dependent. X and Y in Ex5.1-3 are dependent: 1/36=f(1,1) ≠ f1(1)f2(1)=11/36*1/36.

Ex4.1-4: The joint p.m.f. f(x,y)=(x+y)/21, x=1,2,3, y=1,2.

– Then, f1(x)=Σy=1~2(x+y)/21=(2x+3)/21, x=1,2,3.

– Likewise, f2(1)=Σx=1~3(x+y)/21=(6+3y)/21, y=1,2.

– Since f(x,y)≠f1(x)f2(y), X and Y are dependent.

Ex4.1-6: f(x,y)=xy2/13, (x,y)=(1,1),(1,2),(2,2).

55

Quick Dependence ChecksQuick Dependence Checks Practically, “dependence” can be quickly determined if

– The support of X and Y is NOT rectangular, or S is therefore not the product set {(x,y): x S∈ 1, y S∈ 2}, as in Ex4.1-6.

– f(x,y) cannot be factored (separated) into the product of an x-alone expression and a pure y function.

In Ex4.1-4, f(x,y) is a sum, not a product, of x-alone and y-alone functions.

Ex4.1-7: [Probability Histogram for a joint p.m.f.]

66

Mathematical ExpectationMathematical Expectation If u(X1,X2) is a function of two R.V. X1& X2, then

if it exists, is called the mathematical expectation (or expected value) of u(X1,X2).

– The mean of Xi, i=1,2:

– The variance of Xi: Ex4.1-8: A player selects a chip from a bowl having 8 chips:

3 marked (0,0), 2 (1,0), 2 (0,1), 1 (1,1).

77

Probability Density Function JointProbability Density Function Joint Joint Probability Density Function, joint p.d.f., of two continuo

us-type R.V. X & Y, is an integrable function f(x,y):– f(x,y)≥0; ∫y=-∞~∞∫x=-∞~∞f(x,y)dxdy=1;

– P[(X,Y) A]=∫∫∈ Af(x,y)dxdy, for an event A.

Ex4.1-9: X and Y have the joint p.d.f.– A={(x,y): 0<x<1, 0<y<x}.

– The respective marginal p.d.f.s are

X and Y are independent!

88

Independence of Continuous Type R.V.sIndependence of Continuous Type R.V.s Two continuous type R.V. X and Y are independent iff the j

oint p.d.f. factors into the product of their marginal p.d.f.s.

Ex4.1-10: X and Y have the joint p.d.f.– The support S={(x,y): 0≤x≤y≤1}, bounded by x=0, y=1, x=y lines.– The marginal p.d.f.s are

– Various expected values:

X and Y are dependent!

99

Multivariate Hypergeometric DistributionMultivariate Hypergeometric Distribution Ex4.1-11: Of 200 students, 40 have As, 60 Bs; 100 Cs, Ds,

or Fs.– A sample of size 25 is taken at random without replacement.

X1 is the number of A students, X2 is the number of B students, and

25 –X1–X2 is the number of the other students.

– The space S = {(x1,x2): x1,x2≥0, x1+x2≤25}.

– The marginal p.m.f. of X1can be also obtained as:

X1and X2 are dependent!

From the knowledge of the model.

1010

Binomial ⇒ Trinomial DistributionBinomial ⇒ Trinomial Distribution Trinomial Distribution: The experiment is repeated n times.

– The probability p1: perfect, p2: second; p3: defective, p3=1-p1-p2.

– X1: the number of perfect items, X2 for second, X3 for defective.

– The joint p.m.f. is

– X1 is b(n,p1), X2 is b(n,p2); both are dependent.

Ex4.1-13: In manufacturing a certain item,– 95% of the items are good; 4% are “seconds”, and 1% defective.– An inspector observes n=20 items selected at random, counting the

number X of seconds, and the number Y of defectives.– The probability that at least 2 seconds or at least 2 defective items

are found, namely A={(x,y): x≥2 or y≥2}, is

1111

Correlation CoefficientCorrelation Coefficient For two R.V. X1 & X2,

– The mean of Xi, i=1,2:

– The variance of Xi:

– The covariance of X1 & X2 is

– The correlation coefficient of X1& X2 is

Ex4.2-1: X1& X2 have the joint p.m.f.

→

Not a product Dependent!⇒

1212

Insights of the Meaning of Insights of the Meaning of ρρ Among all points in S, ρ tends to be positive if more points are simultaneously above orbelow their respective means with larger probability.

The least-squares regression line is a line passing given (μx,μy) with the best slope b s.t. K(b)=E{[(Y-μy)-b(X-μx)]2} is minimized.– The square of the vertical distance from a point to the line.

– ρ= ±1: K(b)=0 all the points lie on the least-squares regression line.⇒

– ρ= 0: K(b)=σy2, the line is y=μy; X and Y could be independent!!

ρmeasures the amount of linearity in the probability distribution.

1313

ExampleExample Ex4.2-2: Roll a pair of 4-sided die: X is the number of ones,

Y is the number of twos and threes.– The joint p.m.f. is

– The line of best fit is

1414

Independence ⇒ ρ=0Independence ⇒ ρ=0

The converse is not necessarily true! Ex4.2-3: The joint p.m.f. of X and Y is f(x,y)=1/3, (x,y)=(0,1),

(1,0), (2,1).– Obviously, the support is not “rectangular”, so X and Y are dependen

t.

Empirical Data: from n bivariate observations: (xi,yi), i=1..n.– We can compute the sample mean and variance for each variate.– We can also compute the sample correlation coefficient and the sam

ple least squares regression line. (Ref. p.241)

∵independence

1515

Conditional DistributionsConditional Distributions Def.4.3-1: The conditional probability mass function of X,

given that Y=y, is defined by g(x|y)=f(x,y)/f2(y), if f2(y)>0.

– Likewise, h(y|x)=f(x,y)/f1(x), if f1(x)>0.

Ex.4.3-1: X and Y have the joint p.m.f f(x,y)=(x+y)/21, x=1,2,3; y=1,2.– f1(x)=(2x+3)/21, x=1,2,3; f2(y)=(3y+6)/21, y=1,2.

– Thus, given Y=y, the conditional p.m.f. of X is

– When y=1, g(x|1)=(x+1)/9, x=1,2,3; g(1|1):g(2|1):g(3|1)=2:3:4.

– When y=2, g(x|2)=(x+2)/12, x=1,2,3; g(1|2):g(2|2):g(3|2)=3:4:5.

– Similar relationships about h(y|x) can be obtained.

Dependent!

1616

Conditional Mean and VarianceConditional Mean and Variance The conditional mean of Y, given X=x, is The conditional variance of Y, given X=x, is

Ex.4.3-2: [from Ex.4.3-1] X and Y have the joint p.m.f f(x,y)=(x+y)/21, x=1,2,3; y=1,2.

1717

Relationship about Conditional Relationship about Conditional MeanMean

The point (μX,μY) locates on the above two lines, and is their junction.

The product of the slopes is ρ2. The ratio of the slopes is

These relations can derive the unknown from the others known.

1818

ExampleExample Ex.4.3-3: X and Y have the trinomial p.m.f. with n, p1, p2,

p3=1-p1-p2

– They have the marginal p.m.f. b(n, p1), b(n, p2), so

1919

Example for Continuous-type R.V.Example for Continuous-type R.V. Ex4.3-5: [From Ex4.1-10]

⇒The conditional distribution of Y given X=x is U(x,1).[U(a,b) has mean (b+a)/2, and variance (b-a)2/12.]

2020

Transformations of R.V.sTransformations of R.V.s In Section 3.5, the transformation of a single variable X with

f(x) to another Y=v(X), an increasing or decreasing fn, can be done as:

Ex.4.4-1: X: b(n,p), Y=X2, if n=3, p=1/4, then

– What is the transformation u(X/n) leading to a variance free of p?

Taylor’s expansion about p:

Ex: X: b(100,1/4)or b(100,9/10).

Continuous type

Discrete type

When the variance is constant, or free of p,

2121

Multivariate TransformationsMultivariate Transformations When the function Y=u(X) does not have a single-valued inv

erse, it needs to consider possible inverse functions individually.– Each range will be delimited to match the right inverse.

For multivariate, the derivative is replaced by the Jacobian.– Continuous R.V. X1 and X2 have the joint p.d.f. f(x1, x2).

– If has the single-valued inverse

then the joint p.d.f. of Y1 and Y2 is

– [Most difficult] The mapping of the supports are considered.

2222

Transformation to the Transformation to the IndependentIndependent

Ex4.4-2: X1 and X2 have the joint p.d.f. f(x1, x2)=2, 0<x1<x2<1.

– Consider Y1=X1/X2, Y2=X2:

– The mapping of the supports:

– The marginal p.d.f.:

– ∵g(y1,y2)=g1(y1)g2(y2) Y∴ 1,Y2 Independent.

→

2323

Transformation to the DependentTransformation to the Dependent Ex4.4-3: X1 and X2 are indep., each with p.d.f. f(x)=e-x, 0<x<

∞.– Their joint p.d.f. f(x1, x2)= e-x1e-x2, 0<x1<∞, 0<x2<∞.

– Consider Y1=X1-X2, Y2=X1-X2:

– The mapping of the supports:

– The marginal p.d.f.:

– ∵g(y1,y2) ≠g1(y1)g2(y2) Y∴ 1,Y2 Dependent.

→

Double exponential p.d.f.

2424

Beta DistributionBeta Distribution Ex4.4-4: X1 and X2 have indep. Gamma distributions withα,θ

and β, θ. Their joint p.d.f. is

– Consider Y1=X1/(X1+X2), Y2=X1+X2:i.e., X1=Y1Y2, X2=Y2-Y1Y2.

– The marginal p.d.f.:

– ∵g(y1,y2)=g1(y1)g2(y2) Y∴ 1,Y2 Independent.

Beta p.d.f.

Gamma p.d.f.

2525

Another ExampleAnother Example Ex.4.4-5: U: χ2(r1) and V: χ2(r2) are independent.

– The joint p.d.f. of Z and U is

The knowledge of known distributions and their associated integration relationships are useful to derive the distributions of unknown distributions.

χ2(r1+r2)

2626

Two independent R.V.s have the joint p.m.f. = the product of individual p.m.f.s. – Ex: X1 is the number of spots on a fair die.

f1(x1)=1/6, x1=1,2,3,4,5,6. X2is the number of heads on 4 indep. Tosses of a fair coin.

– If X1 and X2 are indep.

If X1 and X2 have the same p.m.f., their joint p.m.f. is f(x1)*f(x2). – This collection of X1and X2is a random sample of size n

=2 from f(x).

P(X1=1,2 & X2=3,4)

2727

Linear Functions of Indep. R.V.sLinear Functions of Indep. R.V.s Suppose a function Y=X1+X2, S1={1,2,3,4,5,6}, S2=

{0,1,2,3,4}. – Y will have the support S={1,2,…,9,10}. – The p.m.f. g(y) of Y is

The mathematical expectation (or expected value) of a function Y=u(X1,X2) is

– If X1and X2 are indep.

2828

Example Example Ex4.5-1: X1 and X2 are two indep. R.V. from casting a die twic

e. – E(X1)=E(X2)=3.5; Var(X1)=Var(X2)=35/12; E(X1X2)=E(X1)E(X2)=12.25;

– E[(X1-3.5)(X2-3.5)]=E(X1-3.5)E(X2-3.5)=0.

– Y=X1+X2 →E(Y)= E(X1)+E(X2)=7;

– Var(Y)=E[(X1+X2-7)2]=Var(X1)+Var(X2)=35/6. The p.m.f. g(y) of Y with S={2,3,4,…,12} is

2929

General Cases General Cases If X1,…,Xn are indep., then their joint p.d.f. is f1(x1) …fn(xn).

– The expected value of the product u1(x1) …un(xn) is the product of the expected values of u1(x1),…, un(xn).

If all these n distributions are the same, the collection of n indep. and identically distributed (iid) random variables, X1,…,Xn, is a random sample of size n from that common distributio

n. Ex4.5-2: X1, X2, X3, are a random sample from a distribution wi

th p.d.f. f(x)=e-x, 0<x<∞. – The joint p.d.f. is

P(0 < X1 < 1,2 < X2 < 4,3 < X3 < 7)

3030

Distributions of Sums of Indep. R.V.sDistributions of Sums of Indep. R.V.s Distributions of the product of indep. R.V.s are strai

ghtforward. However, distributions of the sum of indep. R.V.s ar

e fetching: – First, the joint p.m.f. or p.d.f. is a simple product. – However, through summation, these R.V.s interfere with

each other. Care must be taken to distinguish some sum value happens mor

e frequently than the others.

Sampling distribution theory is to derive the distributions of the functions of R.V.s (random variables). – The sample mean and variance are famous functions. – The summation of R.V.s is another example.

3131

Example Example Ex: X1 and X2 are two indep. R.V.s from casting a 4-sided die t

wice. – The p.m.f. f(x)=1/4, x=1,2,3,4.

– The p.m.f. of Y=X1+X2 with S={2,3,4,5,6,7,8} is g(y):

(convolution formula) g(2)=g(3)=

g(y)=

3232

Theorems Theorems Thm4.5-1: X1,…,Xn are indep. and have the joint p.m.f. is f1(x

1) …fn(xn). Y=u(X0,…,Xn) have the p.m.f. g(y)– Thenif the summations exist. – For continuous type, integrals replace the summations.

Thm4.5-2: X1,…,Xn are indep. and their means exist, – Then,

Thm4.6-1: If X1,…,Xn are indep. with means μ1,…,μn and variances σ1

2,…,σn2, then Y=a1X1+…+anXn, where ai’s are real c

onstants, have the mean and variance:

Ex4.6-1: X1 & X2 are indep. with μ1= -4, μ2=3 and σ12=4, σ2

2=9. – Y=3X1-2X2 has

3333

Moment-generating FunctionsMoment-generating Functions Ex4.6-2: X1,…,Xn are a random sample of size n from a distrib

ution with mean μand variance σ2; then

– The sample mean:

Thm4.6-2: If X1,…,Xn are indep. R.V.s with moment-generating functions , i=1..n, then Y=a1X1+…+anXn, has the moment-generating

Cly4.6-1: If X1,…,Xn are indep. R.V.s with M(t),

– then Y=X1+…+Xn has MY(t)=[M(t)]n.

– has

MY(t)

3434

Examples Examples Ex4.6-3: X1,…,Xn are the outcomes on n Bernoulli tri

als. – The moment-generating function of Xi, i=1..n, is M(t)=q+pet.

– Then Y=X1+…+Xn has MY(t)=[q+pet]n, which is b(n,p).

Ex4.6-4: X1,X2,X3 are the outcomes of a random sample of size n=3 from the exponential distribution with mean θand M(t)=1/(1-θt), t<1/θ.– Then Y=X1+X2+X3 has MY(t)=[1/(1-θt)]3=(1-θt)-3,

which is a gamma distribution with α=3, and θ.– has

a gamma distribution with α=3, and θ/3.

3535

Thm4.6-3: X1,…,Xn are independent and have χ2(r1),…, χ2(rn) distributions, respectively ; Then, Y=X1+…+Xn is χ2(r1+…+rn).

Pf: MY(t)=