2k factorial designs and - rice university
TRANSCRIPT
![Page 1: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/1.jpg)
Dr. John Mellor-Crummey
Department of Computer ScienceRice University
2k Factorial Designs and2kr Designs with Replications
COMP 528 Lecture 12 24 February 2005
![Page 2: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/2.jpg)
2
Goals for Today
Understand• 2k factorial designs
—sign table method—properties—analysis—allocating variation
• 2kr designs with replications—error estimation—confidence intervals for effects
• Multiplicative models
![Page 3: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/3.jpg)
3
2k Factorial Designs
• Determine effects of k factors, each at two levels• Total of 2k experiments required• Total of 2k effects
—k main effects
— two-factor interactions
— three-factor interactions
— j factor interactions, j ≤ k!
k
2
" # $ % & '
!
k
3
" # $ % & '
!
kj" # $ % & '
![Page 4: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/4.jpg)
4
Example: a 23 Design
Evaluate impact of memory, cache size, # processors
Model performance using a non-linear equation in xA and xB
Solve using sign table method
!
y = q0
+ qA xA + qB xB + qAB xA xB + qAC xA xC +
qBC xB xC + qABC xA xB xC
21# processors, C6MB4MBCache Size, B2GB1GBMemory Size, A
Level 1Level -1Factor
![Page 5: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/5.jpg)
5
Calculating Effects with a Sign Table
qABCqBCqACqABqCqBqAq0
32411-1-1-1-111
BC
86111111150-1-1-111-1158-11-11-111461-111-1-11
216
-11-11
AC
540
1-1-11
AB
20160
-1-1-1-1C
total/8151040total84080320
34102214y
-111111-111-111-1-1-11
ABCBAI
![Page 6: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/6.jpg)
6
Calculating Effects
Effects—SSA/SST = 23(102)/4512 = .18 = 18%—SSB/SST = 23(52)/4512 = .04 = 4%—SSC/SST = 23(202)/4512 = .71= 71%—SSAB/SST = 23(52)/4512 = .04 = 4%—SSAC/SST = 23(22)/4512 = .01 = 1%—SSBC/SST = 23(32)/4512 = .02 = 2%—SSABC/SST = 23(12)/4512 = .00 = 0%
!
SST = 23(qA
2+ qB
2+ qC
2+ qAB
2+ qAC
2+ qBC
2+ qABC
2)
!
SST = 23(10
2+ 5
2+ 20
2+ 5
2+ 2
2+ 3
2+1
2) = 4512
qABCqBCqACqABqCqBqAq0
13252051040
![Page 7: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/7.jpg)
7
2kr Factorial Designs with Replications
![Page 8: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/8.jpg)
8
2kr Factorial Designs
• Cannot estimate errors with 2k factorial design—no experiment is repeated
• To quantify experimental errors—repeat measurements with same factor combinations—analyze using sign table
• 2kr design—r replications of 2k experiments (each of 2k factor combinations)
• Model
(e = experimental error)
!
y = q0
+ qA xA + qB xB + qAB xA xB + e
![Page 9: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/9.jpg)
9
Computing Effects for a 22r Design
• Factor combination = sign table row• For each factor combination i, r replicants (here, r = 3)• = mean observation for ith factor combination
• Solve for qi coefficients using sign table and values
qABqBqAq0
total/459.521.541total20388616477244815
1111-11-11-1-1111-1-11
ABBAI
817575192825514845121815
!
y i. yij
!
y i.
!
y i. =1
ryij
j
"
!
y i.
![Page 10: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/10.jpg)
10
Estimating Experimental Errors for a 22r Design
• Model predicts following value for factor combination i
• Errors
• Total error across row is 0 by design• Corollary: total error (across all rows) is also 0
!
ˆ y i = q0
+ qA xAi + qB xBi + qAB xAixBi
!
eij = yij " ˆ y i= yij " q
0+ qA xAi + qB xBi + qAB xAixBi
= yij " y i.
![Page 11: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/11.jpg)
11
Sum of Squared Errors for a 223 Design
!
SSE = eijj=1
r
"i=1
22
"
77244815
1111-11-11-1-1111-1-11
ABBAI
817575192825514845121815
!
y i. yij
4-2-2-54130-3-330
eij
!
SSE = 02 + 32 + "3( )2
+ "3( )2
+ 02 + 32 +
12 + 42 + "5( )
2
+ "2( )2
+ "2( )2
+ 42 =102
![Page 12: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/12.jpg)
12
Mean Response for a 22r Design
• Model for 22r design
• Sum across all 22r observations =
• Since sum of x’s, their products and errors are all 0!
yij = q0
+ qA xAi + qB xBi + qAB xAixBi + eij
!
yiji, j
" = q0
i, j
" + qA xAii, j
" + qB xBii, j
" + qAB xAixBii, j
" + eiji, j
"
!
yiji, j
" = q0
i, j
" = 22rq
0
!
y ..
=1
22r
yij
i, j
" =1
22r22rq
0= q
0 mean response
![Page 13: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/13.jpg)
13
SST and SSE for a 22r Design
!
SST = yij " y ..( )2
=i, j
# yij
2"
i, j
# y ..
2
i, j
#
!
SST = SSY " SS0 = SSA + SSB + SSAB + SSE
!
SSE = SSY " SS0 " SSA " SSB " SSAB
!
SSE = SSY " 22r(q
0
2+ qA
2+ qB
2+ qAB
2)
!
SS0 = y ..
2
i, j
" = q0
2
i, j
"
!
SSY = yij2
i, j
" = q0
2
i, j
" + qA2xAi2
i, j
" + qB2xBi2
i, j
" +
qAB2xAi2xBi2
i, j
" + eij2
i, j
" + product terms (0 by design of sign table)
!
SSY = SS0 + SSA + SSB + SSAB + SSE
![Page 14: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/14.jpg)
14
Allocating Variation for a 22r Design
• Fraction of variation attributed to A = SSA/SST• Fraction of variation attributed to B = SSB/SST• Fraction of variation attributed to interaction of A & B =
SSAB/SST• Fraction of variation attributed to error = SSE/SST
![Page 15: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/15.jpg)
15
Variance for q0
• Effects computed from a sample are random variables• Confidence intervals for effects
—compute from variance of sample estimates
• If errors ~ , model implies that y ~• Consider effect q0
—q0 = linear combination of normally distributed variables—therefore, q0 is normally distributed as well
• Variance of q0
!
N(0,"e
2)
!
N(y ..," e
2)
!
q0
=1
22r
yiji, j
"
!
var(q0) =
" e
2
22r
![Page 16: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/16.jpg)
16
Confidence Intervals for Effects in 22r Designs
• Estimate variance of errors from SSE
• Denominator = 22(r-1) = DOF = # independent terms in SSE—error terms for r replications all add to 0—only r-1 are independent
• Std dev of effects
• Confidence intervals for effects• Effect is significant if its confidence interval does not contain 0
!
se
2=
SSE
22(r "1)
= MSE
!
sq0
= sqA = sqB = sqAB =se
22r
!
qi ± t[1"# / 2;22 (r"1)]
sqi
![Page 17: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/17.jpg)
17
Returning to our Example …
Memory/Cache Study• Standard deviation of errors
• Standard deviation of effects
• Confidence intervals for effects—90% confidence:
• No confidence intervals include 0 ⇒ all significant
!
se
=SSE
22(r "1)
=102
8= 12.75 = 3.57
!
sqi = se / 22r = 3.57 / 12 =1.03
qABqBqAq0
59.521.541
!
qi ± t[1"# / 2;22 (r"1)]
sqi = qi ± (1.86)(1.03) = qi ±1.92
!
t[1"# / 2;22 (r"1)]
= t[.95;8] =1.86
q0= (30.08,42.91), qA= (19.58, 23.41), qB= (7.58, 11.41), qAB= (3.08, 6.91)
![Page 18: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/18.jpg)
18
Assumptions
• Model errors are statistically independent• Errors are additive• Errors are normally distributed• Errors have constant standard deviation• Effects of factors are additive• observations are independent and normally distributed with
constant variance!
"e
![Page 19: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/19.jpg)
19
Visual Tests for Verifying Assumptions
• Independent errors—scatter plot of residuals vs. predicted responses
– should not have any trend– magnitude residuals < magnitude responses/10 ⇒ ignore trends
—plot residuals as a function of experiment number– trend up or down implies other factors or side effects
• Normally distributed errors—normal quantile-quantile plot of errors should be linear
• Constant standard deviation of errors—scatter plot of y for various levels of factors—if spread at one level significantly different than at another
– need transformation
• What to do—if any test fails or ymax/yminis large, investigate multiplicative
model
![Page 20: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/20.jpg)
20
Multiplicative Models for 22r Experiments
• Additive model
• Not applicable if effects don’t add—example: execution time of workloads
– factor A: ith processor speed = vi instructions/sec– factor B: jth workload size = wj instructions
—the two effects multiply: execution time yij = vi x wj
• Use logarithm to produce additive model—log(yij) = log(vi) + log(wj)
• Better modelwhere
!
yij = q0
+ qA xAi + qB xBi + qAB xAixBi + eij
!
y'ij = q0 + qA xAi + qB xBi + qAB xAixBi + eij
!
y'ij = log(yij )
![Page 21: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/21.jpg)
21
Interpreting a Multiplicative Model
• uA is ratio of MIPS rating of two processors• uB ratio of size of the two workloads• Antilog of additive mean q0 yields geometric mean!
uA =10qA ,uB =10
qB ,uAB =10qAB ,
!
˙ y =10q0 = (y
1y
2...yn )
1/ n where n = 22r
!
log(y) = q0 + qA xA + qB xB + qAB xA xB + e
!
y =10q010
qAxA10qBxB10
qABxA xB10e
![Page 22: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/22.jpg)
22
Example: Execution Times
• Analysis using an additive modelfactor A = proc speed; factor B = instructions
qABqBqAq0
total/425.54-26.04-26.0426.55total102.17-104.15-104.15106.190.0131.0031.003
104.170
1111-11-11-1-1111-1-11
ABBAI
.0118.0126.01481.122.933.9551.0721.047.891147.9079.5085.10
!
y i. yij
![Page 23: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/23.jpg)
23
Visual Tests for Additive Model
![Page 24: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/24.jpg)
24
Why is This Model Invalid?
• Physical consideration—workload & processor effects don’t add
• Large range for y— ymax/ymin = 147.90/.0118 = 12,534 ⇒ log transform
– taking arithmetic mean of 104.17 and .013 is inappropriate
• Residuals are not small compared to response• Spread of residuals is larger at larger value of response
—suggests log transformation
![Page 25: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/25.jpg)
25
Analysis using a Multiplicative Model
Data after log transformationfactor A = proc speed; factor B = instructions
qABqBqAq0
total/4.03-0.97-0.97.03total.11-3.89-3.89.11-1.890.000.002.00
1111-11-11-1-1111-1-11
ABBAI
-1.93-1.90-1.83.05-.03-.02.03.02-.052.171.901.93
!
y i. yij
![Page 26: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/26.jpg)
26
Visual Tests for Multiplicative Model
• Conclusion: Multiplicative model is better than additive model
![Page 27: 2k Factorial Designs and - Rice University](https://reader034.vdocuments.mx/reader034/viewer/2022051000/627676ad60c8f75f1a7ed57d/html5/thumbnails/27.jpg)
27
Variation Explained by the Models
Additive Multiplicative
• With multiplicative model: interaction almost 0• Little unexplained variation: .2%
.2%10.8%e(-.02,.07)*0.0%.03(15.35,35.74)29.0%25.54AB
(-1.02,-.93)49.9%-.97(-36.23,-15.84)30.1%-26.04B
(-1.02,-.93)49.9%-.97(-36.23,-15.84)30.1%-26.04A
(-.02,.07)*.03(16.35,36.74)26.55I
ConfInterval
%varEffectConfInterval
%varEffectFactor
* = not significant