introduction to the gauss-markov linear modeldnett/s511/01introduction.pdf · the gauss-markov...
TRANSCRIPT
Introduction to the Gauss-Markov LinearModel
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 1 / 36
Random Vectors
y =
y1y2...
yn
is a random vector if and only if each element of y is a
random variable (i.e., yi is a random variable ∀ i = 1, . . . , n).
The mean of the random vector y is E(y) =
E(y1)E(y2)
...E(yn)
.
The variance of the random vector y is the matrix whose i, jthelement is Cov(yi, yj) = E(yiyj)− E(yi)E(yj).
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 2 / 36
Example: Variance of a Random Vector
For example, the variance of y =
y1
y2
y3
is
Var(y) =
Cov(y1, y1) Cov(y1, y2) Cov(y1, y3)Cov(y2, y1) Cov(y2, y2) Cov(y2, y3)Cov(y3, y1) Cov(y3, y2) Cov(y3, y3)
=
Var(y1) Cov(y1, y2) Cov(y1, y3)Cov(y2, y1) Var(y2) Cov(y2, y3)Cov(y3, y1) Cov(y3, y2) Var(y3)
.
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 3 / 36
The Gauss-Markov Linear Model
y = Xβ + ε
y is an n× 1 random vector of responses.
X is an n× p matrix of constants with columns corresponding toexplanatory variables. X is sometimes referred to as the designmatrix.
β is an unknown parameter vector in IRp.
ε is an n× 1 random vector of errors.
E(ε) = 0 and Var(ε) = σ2I, where σ2 is an unknown parameter inIR+.
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 4 / 36
The Gauss-Markov Linear Model
Note that the model is not completely specified because thedistribution of y is not completely specified.
y = Xβ + ε, E(ε) = 0, Var(ε) = σ2I
=⇒ E(y) = Xβ, Var(y) = σ2I=⇒ y ∼ (Xβ, σ2I)
“y has a distribution with mean Xβ and variance σ2I.”
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 5 / 36
The Normal Theory Gauss-Markov Linear Model
We often add an assumption of multivariate normality to theGauss-Markov linear model: ε ∼ N(0, σ2I).
The assumption ε ∼ N(0, σ2I) is equivalent toε1, . . . , εn
i.i.d.∼ N(0, σ2).
The assumption ε ∼ N(0, σ2I) =⇒ y ∼ N(Xβ, σ2I), i.e.,y1, . . . , yn are independent normal random variables,
Var(yi) = σ2 ∀ i = 1, . . . , n, and
E(yi) = x′(i)β (where x′(i) is the ith row of X) ∀ i = 1, . . . , n.
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 6 / 36
Goal of Analysis
y = Xβ + ε
The goal of analysis often focuses on answering questionsabout certain linear functions of β of the form Cβ for aspecified matrix C.
The normality assumption is useful for constructingconfidence intervals and performing tests concerning Cβ.
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 7 / 36
Example 1Researchers harvested five randomly selected ears of corn from afield. For i = 1, . . . , 5; let yi denote the weight in grams of the ith ear.
y1, . . . , y5i.i.d.∼ N(µ, σ2)
yi = µ+ εi, i = 1, . . . , 5; ε1, . . . , ε5i.i.d.∼ N(0, σ2)
y1 = µ+ ε1
y2 = µ+ ε2
y3 = µ+ ε3 ε1, . . . , ε5i.i.d.∼ N(0, σ2)
y4 = µ+ ε4
y5 = µ+ ε5
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 8 / 36
Example 1 (continued)
y1 = µ+ ε1
y2 = µ+ ε2
y3 = µ+ ε3 ε1, . . . , ε5i.i.d.∼ N(0, σ2)
y4 = µ+ ε4
y5 = µ+ ε5
y1y2y3y4y5
=
µµµµµ
+
ε1ε2ε3ε4ε5
,ε1ε2ε3ε4ε5
∼ N(0, σ2I)
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 9 / 36
Example 1 (continued)
y1y2y3y4y5
=
µµµµµ
+
ε1ε2ε3ε4ε5
,ε1ε2ε3ε4ε5
∼ N(0, σ2I)
y1y2y3y4y5
=
11111
[µ] +
ε1ε2ε3ε4ε5
,ε1ε2ε3ε4ε5
∼ N(0, σ2I)
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 10 / 36
Example 1 (continued)
y1y2y3y4y5
=
11111
[µ] +
ε1ε2ε3ε4ε5
,ε1ε2ε3ε4ε5
∼ N(0, σ2I)
y = Xβ + ε, ε ∼ N(0, σ2I)
Cβ = [1][µ] = µ
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 11 / 36
Example 2
Researchers randomly assigned eight experimental units to twotreatments and measured a response of interest. For i = 1, 2; letyi1, yi2, yi3, yi4 denote the responses of the experimental units in the ith
treatment group.
y11, y12, y13, y14i.i.d.∼ N(µ1, σ
2)
independent of
y21, y22, y23, y24i.i.d.∼ N(µ2, σ
2)
yij = µi + εij, i = 1, 2; j = 1, . . . , 4
ε11, ε12, ε13, ε14, ε21, ε22, ε23, ε24i.i.d.∼ N(0, σ2)
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 12 / 36
Example 2 (continued)
y11 = µ1 + ε11
y12 = µ1 + ε12
y13 = µ1 + ε13
y14 = µ1 + ε14
y21 = µ2 + ε21
y22 = µ2 + ε22
y23 = µ2 + ε23
y24 = µ2 + ε24
ε11, ε12, ε13, ε14, ε21, ε22, ε23, ε24i.i.d.∼ N(0, σ2)
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 13 / 36
Example 2 (continued)
y11y12y13y14y21y22y23y24
=
µ1µ1µ1µ1µ2µ2µ2µ2
+
ε11ε12ε13ε14ε21ε22ε23ε24
,
ε11ε12ε13ε14ε21ε22ε23ε24
∼ N(0, σ2I)
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 14 / 36
Example 2 (continued)
y11y12y13y14y21y22y23y24
=
1 01 01 01 00 10 10 10 1
[µ1µ2
]+
ε11ε12ε13ε14ε21ε22ε23ε24
,
ε11ε12ε13ε14ε21ε22ε23ε24
∼ N(0, σ2I)
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 15 / 36
Example 2 (continued)
y11y12y13y14y21y22y23y24
=
1 01 01 01 00 10 10 10 1
[µ1µ2
]+
ε11ε12ε13ε14ε21ε22ε23ε24
,
ε11ε12ε13ε14ε21ε22ε23ε24
∼ N(0, σ2I)
y = Xβ + ε, ε ∼ N(0, σ2I)
Cβ = [1,−1][µ1µ2
]= µ1 − µ2
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 16 / 36
Example 3Suppose eight fertilizer amounts denoted x1, . . . , x8 were randomlyassigned to eight field plots. For i = 1, . . . , 8; let yi denote the yield ofthe plot that received fertilizer amount xi.
yi = β0 + β1xi + εi, i = 1, . . . , 8
ε1, . . . , ε8i.i.d.∼ N(0, σ2)
y1 = β0 + β1x1 + ε1
y2 = β0 + β1x2 + ε2
y3 = β0 + β1x3 + ε3
y4 = β0 + β1x4 + ε4
y5 = β0 + β1x5 + ε5
y6 = β0 + β1x6 + ε6
y7 = β0 + β1x7 + ε7
y8 = β0 + β1x8 + ε8Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 17 / 36
Example 3 (continued)
y1y2y3y4y5y6y7y8
=
β0 + β1x1β0 + β1x2β0 + β1x3β0 + β1x4β0 + β1x5β0 + β1x6β0 + β1x7β0 + β1x8
+
ε1ε2ε3ε4ε5ε6ε7ε8
,
ε1ε2ε3ε4ε5ε6ε7ε8
∼ N(0, σ2I)
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 18 / 36
Example 3 (continued)
y1y2y3y4y5y6y7y8
=
1 x11 x21 x31 x41 x51 x61 x71 x8
[β0β1
]+
ε1ε2ε3ε4ε5ε6ε7ε8
,
ε1ε2ε3ε4ε5ε6ε7ε8
∼ N(0, σ2I)
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 19 / 36
Example 3 (continued)
y1y2y3y4y5y6y7y8
=
1 x11 x21 x31 x41 x51 x61 x71 x8
[β0β1
]+
ε1ε2ε3ε4ε5ε6ε7ε8
,
ε1ε2ε3ε4ε5ε6ε7ε8
∼ N(0, σ2I)
y = Xβ + ε, ε ∼ N(0, σ2I)
Cβ = [0, 1][β0β1
]= β1
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 20 / 36
Example 4
Eight hogs were randomly assigned to two diets and two inoculationssuch that two hogs received each combination of diet and inoculation.
This experiment involves two factors: diet and inoculation.In this case, each factor has two levels (denoted here genericallyas 1 and 2).A combination of one level from each factor forms a treatment.In this case, we have four treatments:
Treatment Diet Inoculation1 1 12 1 23 2 14 2 2
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 21 / 36
Example 4 (continued)
For i = 1, 2; j = 1, 2; and k = 1, 2; let yijk denote the average daily gainof the kth hog that received diet i and inoculation j.
yijk = µ+ εijk i = 1, 2; j = 1, 2; k = 1, 2;
ε111, ε112, ε121, ε122, ε211, ε212, ε221, ε222i.i.d.∼ N(0, σ2)
Under this model, neither diet nor inoculation affects average dailygain.
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 22 / 36
Example 4 (continued)
For i = 1, 2; j = 1, 2; and k = 1, 2; let yijk denote the average daily gainof the kth hog that received diet i and inoculation j.
yijk = µ+ αi + εijk i = 1, 2; j = 1, 2; k = 1, 2;
ε111, ε112, ε121, ε122, ε211, ε212, ε221, ε222i.i.d.∼ N(0, σ2)
Under this model, only diet affects average daily gain.
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 23 / 36
Example 4 (continued)
For i = 1, 2; j = 1, 2; and k = 1, 2; let yijk denote the average daily gainof the kth hog that received diet i and inoculation j.
yijk = µ+ βj + εijk i = 1, 2; j = 1, 2; k = 1, 2;
ε111, ε112, ε121, ε122, ε211, ε212, ε221, ε222i.i.d.∼ N(0, σ2)
Under this model, only inoculation affects average daily gain.
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 24 / 36
Example 4 (continued)
yijk = µ+ αi + βj + εijk i = 1, 2; j = 1, 2; k = 1, 2;
ε111, ε112, ε121, ε122, ε211, ε212, ε221, ε222i.i.d.∼ N(0, σ2)
Under this model, factors diet and inoculation affect the meanaverage daily gain in an additive manner.There is no interaction between the factors diet and inoculation.
inoculationdiet 1 2 inoculation difference1 µ+ α1 + β1 µ+ α1 + β2 β1 − β22 µ+ α2 + β1 µ+ α2 + β2 β1 − β2
diet difference α1 − α2 α1 − α2
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 25 / 36
Example 4 (continued)
yijk = µ+ αi + βj + γij + εijk i = 1, 2; j = 1, 2; k = 1, 2;
ε111, ε112, ε121, ε122, ε211, ε212, ε221, ε222i.i.d.∼ N(0, σ2)
Under this model, there is one mean for each combination of dietand inoculation.Those four means are free to take any four values with norestrictions.
inoculationdiet 1 2 ∆inoculation1 µ+ α1 + β1 + γ11 µ+ α1 + β2 + γ12 β1 − β2 + γ11 − γ122 µ+ α2 + β1 + γ21 µ+ α2 + β2 + γ22 β1 − β2 + γ21 − γ22
∆diet α1 − α2 + γ11 − γ21 α1 − α2 + γ12 − γ22
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 26 / 36
Example 4 (continued)
An equivalent model is the so called cell means model:
yijk = µij + εijk i = 1, 2; j = 1, 2; k = 1, 2;
ε111, ε112, ε121, ε122, ε211, ε212, ε221, ε222i.i.d.∼ N(0, σ2)
inoculationdiet 1 2 ∆inoculation1 µ11 µ12 µ11 − µ122 µ21 µ22 µ21 − µ22
∆diet µ11 − µ21 µ12 − µ22
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 27 / 36
Example 4 (continued)
yijk = µ+ αi + βj + γij + εijk i = 1, 2; j = 1, 2; k = 1, 2;
y111 = µ+ α1 + β1 + γ11 + ε111
y112 = µ+ α1 + β1 + γ11 + ε112
y121 = µ+ α1 + β2 + γ12 + ε121
y122 = µ+ α1 + β2 + γ12 + ε122
y211 = µ+ α2 + β1 + γ21 + ε211
y212 = µ+ α2 + β1 + γ21 + ε212
y221 = µ+ α2 + β2 + γ22 + ε221
y222 = µ+ α2 + β2 + γ22 + ε222
ε111, ε112, ε121, ε122, ε211, ε212, ε221, ε222i.i.d.∼ N(0, σ2)
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 28 / 36
Example 4 (continued)
y111y112y121y122y211y212y221y222
=
µ+ α1 + β1 + γ11µ+ α1 + β1 + γ11µ+ α1 + β2 + γ12µ+ α1 + β2 + γ12µ+ α2 + β1 + γ21µ+ α2 + β1 + γ21µ+ α2 + β2 + γ22µ+ α2 + β2 + γ22
+
ε111ε112ε121ε122ε211ε212ε221ε222
,
ε111ε112ε121ε122ε211ε212ε221ε222
∼ N(0, σ2I)
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 29 / 36
Example 4 (continued)
y111y112y121y122y211y212y221y222
=
1 1 0 1 0 1 0 0 01 1 0 1 0 1 0 0 01 1 0 0 1 0 1 0 01 1 0 0 1 0 1 0 01 0 1 1 0 0 0 1 01 0 1 1 0 0 0 1 01 0 1 0 1 0 0 0 11 0 1 0 1 0 0 0 1
µα1α2β1β2γ11γ12γ21γ22
+
ε111ε112ε121ε122ε211ε212ε221ε222
y = Xβ + ε, ε ∼ N(0, σ2I)
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 30 / 36
Example 4 (continued)
β = [µ, α1, α2, β1, β2, γ11, γ12, γ21, γ22]′
inoculationdiet 1 21 µ+ α1 + β1 + γ11 µ+ α1 + β2 + γ122 µ+ α2 + β1 + γ21 µ+ α2 + β2 + γ22
∆diet α1 − α2 + γ11 − γ21 α1 − α2 + γ12 − γ22
Is the difference between diet means for inoculation 1 the same as thedifference between diet means for inoculation 2?
Cβ = [0, 0, 0, 0, 0, 1,−1,−1, 1]β = γ11 − γ12 − γ21 + γ22 = 0?
This questions asks if there is interaction between the factors diet andinoculation.
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 31 / 36
Example 4 (continued)
β = [µ, α1, α2, β1, β2, γ11, γ12, γ21, γ22]′
inoculationdiet 1 2 ∆inoculation1 µ+ α1 + β1 + γ11 µ+ α1 + β2 + γ12 β1 − β2 + γ11 − γ122 µ+ α2 + β1 + γ21 µ+ α2 + β2 + γ22 β1 − β2 + γ21 − γ22
Is the difference between inoculation means for diet 1 the same as thedifference between inoculation means for diet 2?
Cβ = [0, 0, 0, 0, 0, 1,−1,−1, 1]β = γ11 − γ12 − γ21 + γ22 = 0?
This questions also asks if there is interaction between the factors dietand inoculation.
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 32 / 36
Example 4 (continued)
β = [µ, α1, α2, β1, β2, γ11, γ12, γ21, γ22]′
inoculationdiet 1 2 Diet Means1 µ+ α1 + β1 + γ11 µ+ α1 + β2 + γ12 µ+ α1 + β· + γ1·2 µ+ α2 + β1 + γ21 µ+ α2 + β2 + γ22 µ+ α2 + β· + γ2·
Is the average over inoculation means for diet 1 different than theaverage over inoculation means for diet 2?
Cβ = [0, 1,−1, 0, 0, .5, .5,−.5,−.5]β = α1 − α2 + γ1· − γ2· = 0?
This question asks about the main effect of the factor diet.
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 33 / 36
Example 4 (continued)
β = [µ, α1, α2, β1, β2, γ11, γ12, γ21, γ22]′
inoculationdiet 1 21 µ+ α1 + β1 + γ11 µ+ α1 + β2 + γ122 µ+ α2 + β1 + γ21 µ+ α2 + β2 + γ22
Inoculation Means µ+ α· + β1 + γ·1 µ+ α· + β2 + γ·2
Is the average over diet means for inoculation 1 different than theaverage over diet means for inoculation 2?
Cβ = [0, 0, 0, 1,−1, .5,−.5, .5,−.5]β = β1 − β2 + γ·1 − γ·2 = 0?
This question asks about the main effect of the factor inoculation.
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 34 / 36
Example 4 (continued)
β = [µ, α1, α2, β1, β2, γ11, γ12, γ21, γ22]′
inoculationdiet 1 21 µ+ α1 + β1 + γ11 µ+ α1 + β2 + γ122 µ+ α2 + β1 + γ21 µ+ α2 + β2 + γ22
∆diet α1 − α2 + γ11 − γ21
Is there a difference between the diet means for inoculation 1?
Cβ = [0, 1,−1, 0, 0, 1, 0,−1, 0]β = α1 − α2 + γ11 − γ21 = 0?
This question asks about the simple effect of the factor diet for the firstlevel of the factor inoculation.
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 35 / 36
Example 4 (continued)
β = [µ, α1, α2, β1, β2, γ11, γ12, γ21, γ22]′
inoculationdiet 1 2 ∆inoculation1 µ+ α1 + β1 + γ11 µ+ α1 + β2 + γ12 β1 − β2 + γ11 − γ122 µ+ α2 + β1 + γ21 µ+ α2 + β2 + γ22 β1 − β2 + γ21 − γ22
∆diet α1 − α2 + γ11 − γ21 α1 − α2 + γ12 − γ22
Are all four treatment means identical?
Cβ =
0 0 0 1 −1 1 −1 0 00 0 0 1 −1 0 0 1 −10 1 −1 0 0 1 0 −1 0
β
=
β1 − β2 + γ11 − γ12β1 − β2 + γ21 − γ22α1 − α2 + γ11 − γ21
=
000
?
Copyright c©2012 Dan Nettleton (Iowa State University) Statistics 511 36 / 36