ratio games and designing experiments
DESCRIPTION
Ratio Games and Designing Experiments. Andy Wang CIS 5930-03 Computer Systems Performance Analysis. Ratio Games. Choosing a base system Using ratio metrics Relative performance enhancement Ratio games with percentages Strategies for winning a ratio game Correct analysis of ratios. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/1.jpg)
Ratio Gamesand
Designing ExperimentsAndy WangCIS 5930-03
Computer SystemsPerformance Analysis
![Page 2: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/2.jpg)
2
Ratio Games• Choosing a base system• Using ratio metrics• Relative performance enhancement• Ratio games with percentages• Strategies for winning a ratio game• Correct analysis of ratios
![Page 3: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/3.jpg)
3
Choosing a Base System
• Run workloads on two systems• Normalize performance to chosen
system• Take average of ratios• Presto: you control what’s best
![Page 4: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/4.jpg)
4
Example of Choosinga Base System
• (Carefully) selected Ficus results:
1 2 1/2 2/1cp 179.5 168.6 1.06 0.94rcp 286.8 338.3 0.85 1.18Mean 233.2 253.5 0.96 1.06
![Page 5: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/5.jpg)
5
Why Does This Work?• Expand the arithmetic:
abbi
ai
b
a
b
abaiba
bbb
aba
Pyn
yn
yy
yy
nR
nP
RyyR
;;
;
;1
;1
;0
;0;;;
;;
11
1
11
0.1
![Page 6: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/6.jpg)
6
Using Ratio Metrics• Pick a metric that is itself a ratio
– E.g., power = throughput response time– Or cost/performance
• Handy because division is “hidden”
![Page 7: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/7.jpg)
7
Relative Performance Enhancement• Compare systems with incomparable bases
• Turn into ratios• Example: compare Ficus 1 vs. 2 replicas
with UFS vs. NFS (1 run on chosen day):
• “Proves” adding Ficus replica costs less than going from UFS to NFS
"cp" Time RatioFicus 1 vs. 2 197.4 246.6 1.25UFS vs. NFS 178.7 238.3 1.33
![Page 8: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/8.jpg)
8
Ratio Games with Percentages
• Percentages are inherently ratios– But disguised– So great for ratio games
• Example: Passing tests
• A is worse, but looks better in total line!
Test A Runs A Passes A % B RunsB PassesB %1 300 60 20 32 8 252 50 2 4 500 40 8
Total 350 62 18 532 48 9
![Page 9: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/9.jpg)
9
More on Percentages• Psychological impact
– 1000% sounds bigger than 10-fold (or 11-fold)
– Great when both original and final performance are lousy
• E.g., salary went from $40 to $80 per week
• Small sample sizes generate big lies• Base should be initial, not final value
– E.g., price can’t drop 400%
![Page 10: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/10.jpg)
10
Strategies for Winning
a Ratio Game• Can you win?• How to win
![Page 11: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/11.jpg)
11
Can You Winthe Ratio Game?
• If one system is better by all measures, a ratio game won’t work– But recall percent-passes example– And selecting the base lets you change the
magnitude of the difference• If each system wins on some measures,
ratio games might be possible (but no promises)– May have to try all bases
![Page 12: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/12.jpg)
12
How to WinYour Ratio Game
• For LB metrics, use your system as the base
• For HB metrics, use the other as a base• If possible, adjust lengths of
benchmarks– Elongate when your system performs best– Short when your system is worst– This gives greater weight to your strengths
![Page 13: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/13.jpg)
13
Correct Analysis of Ratios
• Previously covered in Lecture 2• Generally, harmonic or geometric
means are appropriate– Or use only the raw data
![Page 14: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/14.jpg)
14
Introduction To Experimental Design
• You know your metrics• You know your factors• You know your levels• You’ve got your instrumentation and
test loads• Now what?
![Page 15: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/15.jpg)
15
Goals in Experiment Design
• Obtain maximum information with minimum work– Typically meaning minimum number of
experiments• More experiments aren’t better if you’re
the one who has to perform them• Well-designed experiments are also
easier to analyze
![Page 16: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/16.jpg)
16
Experimental Replications
• System under study will be run with varying levels of different factors, potentially with differing workloads
• Run with particular set of levels and other inputs is a replication
• Often, need to do multiple replications with each set of levels and other inputs– Usually necessary for statistical validation
![Page 17: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/17.jpg)
17
Interacting Factors• Some factors have completely
independent effects– Double the factor’s level, halve the
response, regardless of other factors• But effects of some factors depends on
values of others– Called interacting factors
• Presence of interacting factors complicates experimental design
![Page 18: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/18.jpg)
18
The Basic Problemin Designing Experiments
• You’ve chosen some number of factors– May or may not interact
• How to design experiment that captures full range of levels?– Want minimum amount of work
• Which combination or combinations of levels (of factors) do you measure?
![Page 19: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/19.jpg)
19
Common Mistakesin Experimentation
• Ignoring experimental error• Uncontrolled parameters• Not isolating effects of different factors• One-factor-at-a-time experimental
designs• Interactions ignored• Designs require too many experiments
![Page 20: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/20.jpg)
20
Types ofExperimental Designs
• Simple designs• Full factorial design• Fractional factorial design
![Page 21: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/21.jpg)
21
Simple Designs• Vary one factor at a time• For k factors with ith factor having ni
levels, no. of experiments needed is:
• Assumes factors don’t interact– Even then, more effort than required
• Don’t use it, usually
k
iinn
1
11
![Page 22: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/22.jpg)
22
Full Factorial Designs• Test every possible combination of
factors’ levels• For k factors with ith factor having ni levels:
• Captures full information about interaction• But a huge amount of work
k
iinn
1
![Page 23: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/23.jpg)
23
Reducing the Work inFull Factorial Designs
• Reduce number of levels per factor– Generally good choice– Especially if you know which factors are
most important• Use more levels for those
• Reduce number of factors– But don’t drop important ones!
• Use fractional factorial designs
![Page 24: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/24.jpg)
24
Fractional Factorial Designs
• Only measure some combination of levels of the factors
• Must design carefully to best capture any possible interactions
• Less work, but more chance of inaccuracy
• Especially useful if some factors are known to not interact
![Page 25: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/25.jpg)
25
2k Factorial Designs• Used to determine effect of k factors
– Each with two alternatives or levels• Often used as preliminary to larger
performance study– Each factor measured at its maximum and
minimum level– Might offer insight on importance and
interaction of various factors
![Page 26: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/26.jpg)
26
Unidirectional Effects• Effects that only increase as level of a
factor increases– Or vice versa
• If system known to have unidirectional effects, 2k factorial design at minimum and maximum levels is useful
• Shows whether factor has significant effect
![Page 27: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/27.jpg)
27
22 Factorial Designs• Two factors with two levels each• Simplest kind of factorial experiment
design• Concepts developed here generalize• Regression can easily be used
![Page 28: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/28.jpg)
28
22 Factorial Design Example
• Consider parallel operating system• Goal is fastest possible completion of a
given program• Quality usually expressed as speedup• We’ll use runtime as metric (simpler but
equivalent)
![Page 29: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/29.jpg)
29
Factors and Levelsfor Parallel OS
• First factor: number of CPUs– Vary between 8 and 64
• Second factor: use of dynamic load management– Migrates work between nodes as load
changes• Other factors possible, but ignore them
for now
![Page 30: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/30.jpg)
30
Defining Variables for22 Factorial OS Example
11
Ax if 8 nodes
if 64 nodes
11
Bx if no dynamic load mgmt
if dynamic load mgmt
![Page 31: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/31.jpg)
31
Sample Datafor Parallel OS
• Single runs of one benchmark
DLM
NO
DLM
8 Nodes 64 Nodes
820
776 197
217
![Page 32: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/32.jpg)
32
Regression Modelfor Example
• y = q0 + qAxA + qBxB + qABxAxB
• Note that model is nonlinear!
820 = q0 – qA – qB + qAB
217 = q0 + qA – qB – qAB
776 = q0 – qA + qB – qAB
197 = q0 + qA + qB + qAB
![Page 33: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/33.jpg)
33
Solving the Equations• 4 equations in 4 unknowns• q0 = 502.5• qA = -295.5• qB = -16• qAB = 6• So y = 502.5 – 295.5xA – 16xB + 6xAxB
![Page 34: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/34.jpg)
34
The Sign Table Method• Write problem in tabular formI A B AB y1 -1 -1 1 8201 1 -1 -1 2171 -1 1 -1 7761 1 1 1 1972010 -1182 -64 24 Total502.5 -295.5 -16 6 Total/4
![Page 35: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/35.jpg)
35
Allocation of Variationfor 22 Model
• Calculate the sample variance of y:
• Numerator is SST: total variationSST = 22qA
2 + 22qB2 + 22qAB
2 • SST explains causes of variation in y
122
2
1
22
2
i i
y
yys
![Page 36: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/36.jpg)
36
Terms in the SST• 22qA
2 is variation explained by effect of A: SSA
• 22qB2 is variation explained by effect of
B: SSB• 22qAB
2 is variation explained by interaction between A and B: SSAB
SST = SSA + SSB + SSAB
![Page 37: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/37.jpg)
37
Variations in Our Example
• SST = 350449• SSA = 349281• SSB = 1024• SSAB = 144• Now easy to calculate fraction of total
variation caused by each effect
![Page 38: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/38.jpg)
38
Fractions of Variation in Our Example
• Fraction explained by A is 99.67%• Fraction explained by B is 0.29%• Fraction explained by interaction of A
and B is 0.04%• So almost all variation comes from
number of nodes• If you want to run faster, apply more
nodes, don’t turn on dynamic load management
![Page 39: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/39.jpg)
39
General 2k
Factorial Designs• Used to explain effects of k factors,
each with two alternatives or levels• 22 factorial designs are a special case• Same methods extend to more general
case• Many more interactions between pairs
(and trios, etc.) of factors
![Page 40: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/40.jpg)
40
Sample 23 Experiment• Sign table A, B, C are binary count;
interactions are products of columns:I A B C AB AC BC ABC y1 -1 -1 -1 1 1 1 -1 141 1 -1 -1 -1 -1 1 1 221 -1 1 -1 -1 1 -1 1 101 1 1 -1 1 -1 -1 -1 341 -1 -1 1 1 -1 -1 1 461 1 -1 1 -1 1 -1 -1 581 -1 1 1 -1 -1 1 -1 501 1 1 1 1 1 1 1 86
40 10 5 20 5 2 3 1 T/818 4.4 71 4.4 0.7 1.6 0.2 %
![Page 41: Ratio Games and Designing Experiments](https://reader036.vdocuments.mx/reader036/viewer/2022062411/568167d1550346895ddd22fc/html5/thumbnails/41.jpg)
White Slide