simulation modeling and analysis session 12 comparing alternative system designs
Post on 20-Dec-2015
219 views
TRANSCRIPT
![Page 1: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/1.jpg)
Simulation Modeling and Analysis
Session 12
Comparing Alternative System Designs
![Page 2: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/2.jpg)
Outline
• Comparing Two Designs
• Comparing Several Designs
• Statistical Models
• Metamodeling
![Page 3: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/3.jpg)
Comparing two designs
• Let the average measures of performance for designs 1 and 2 be 1 and 2.
• Goal of the comparison: Find point and interval estimates for 1 - 2
![Page 4: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/4.jpg)
Example• Auto inspection system design
• Arrivals: E(6.316) min
• Service: – Brake check N(6.5,0.5) min– Headlight check N(6,0.5) min– Steering check N(5.5,0.5) min
• Two alternatives: – Same service person does all checks– A service person is devoted to each check
![Page 5: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/5.jpg)
Comparing Two Designs -contd
• Run length (ith design ) = Tei
• Number of replications (ith design ) = Ri
• Average response time for replication r (ith design = Yri
• Averages and standard deviations over all replications, Y1* = Yri / Ri and Y2* , are unbiased estimators of 1 and 2.
![Page 6: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/6.jpg)
Possible outcomes• Confidence interval for 1 - 2 well to the
left of zero. I.e. most likely 1 < 2.
• Confidence interval for 1 - 2 well to the right of zero. I.e. most likely 1 > 2.
• Confidence interval for 1 - 2 contains zero. I.e. most likely 1 ~ 2.
• Confidence interval
(Y1* - Y2*) ± t /2, s.e.(Y1* - Y2*)
![Page 7: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/7.jpg)
Independent Sampling with Equal Variances
• Different and independent random number streams are used to simulate the two designs.
Var(Yi*) = var(Yri)/Ri = i2/Ri
Var(Y1* - Y2*) = var(Y1*) + var(Y2*)
= 12/R1 + 2
2/R2 = VIND
• Assume the run lengths can be adjusted to produce 1
2 ~ 22
![Page 8: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/8.jpg)
Independent Sampling with Equal Variances -contd
• Then Y1* - Y2* is a point estimate of 1 - 2
Si2 = (Yri - Yi*)2/(Ri - 1)
Sp2 = [(R1-1) S1
2 + (R2-1) S22]/(R1+R2-2)
s.e.(Y1*-Y2*) = Sp (1/R1 + 1/R2)1/2
= R1 + R2 -2
![Page 9: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/9.jpg)
Independent Sampling with Unequal Variances
s.e.(Y1*-Y2*) = (S12/R1 + S2
2/R2)1/2
= (S12/R1 + S2
2/R2)2/M
where
M = (S12/R1)2/(R1-1) + (S2
2/R2)2/(R2-1)
• Here R1 and R2 must be > 6
![Page 10: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/10.jpg)
Correlated Sampling• Correlated sampling induces positive
correlation between Yr1 and Yr2 and reduces the variance in the point estimator of Y1*-Y2*
• Same random number streams used for both systems for each replication r (R1 = R2 = R)
• Estimates Yr1 and Yr2 are correlated but Yr1 and Ys2 (r n.e. s) are mutually independent.
![Page 11: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/11.jpg)
Recall: Covariance
var(Y1* - Y2*) = var(Y1* ) + var(Y2* ) -
2 cov(Y1* , Y2* ) =
12/R + 2
2/R - 2 12 1 2/R = VCORR
= VIND - 2 12 1 2/R Recall: definition of covariance
cov(X1,X2) = E(X1 X2) - 1 2 =
= corr(X1 X2) 1 2 =
= 1 2
![Page 12: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/12.jpg)
Correlated Sampling -contd• Let Dr =Yr1 - Yr2
D* = (1/R) Dr = Y1* - Y2*
SD2 = (1/(R - 1)) (Dr - D*)2
• Standard error for the 100(1- )% confidence interval
s.e.(D*) = s.e.(Y1* - Y2* ) = SD/ R
(Y1* - Y2*) ± t /2, SD/ R
![Page 13: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/13.jpg)
Correlated Sampling -contd
• Random Number Synchronization Guides– Dedicate a r.n. stream for a specific purpose
and use as many streams as needed. Assign independent seeds to each stream at the beginning of each run.
– For cyclic task subsystems assign a r.n. stream.– If synchronization is not possible for a
subsystem use an independent stream.
![Page 14: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/14.jpg)
Example: Auto inspection
An = interarrival time for vehicles n,n+1
Sn(1) = brake inspection time for vehicle n in model 1
Sn(2) = headlight inspection time for vehicle n in model 1
Sn(3) = steering inspection time for vehicle n in model 1
• Select R = 10, Total_time = 16 hrs
![Page 15: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/15.jpg)
Example: Auto inspection
• Independent runs
-18.1 < 1-2 < 7.3
• Correlated runs
-12.3 < 1-2 < 8.5
• Synchronized runs
-0.5 < 1-2 < 1.3
![Page 16: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/16.jpg)
Confidence Intervals with Specified Precision
• Here the problem is to determine the number of replications R required to achieve a desired level of precision in the confidence interval, based on results obtained using Ro replications
R = (t /2,Ro SD/
![Page 17: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/17.jpg)
Comparing Several System Designs
• Consider K alternative designs
• Performance measure i
• Procedures– Fixed sample size– Sequential sampling (multistage)
![Page 18: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/18.jpg)
Comparing Several System Designs -contd
• Possible Goals– Estimation of each i
– Comparing i to a control 1
– All possible comparisons
– Selection of the best i
![Page 19: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/19.jpg)
Bonferroni Method for Multiple Comparisons
• Consider C confidence intervals 1-i
• Overall error probability E = j
• Probability all statements are true (the parameter is contained inside all C.I.’s)
P 1 - E
• Probability one or more statements are false
P E
![Page 20: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/20.jpg)
Example: Auto inspection (contd)
• Alternative designs for addition of one holding space– Parallel stations– No space between stations in series– One space between brake and headlight
inspection– One space between headlight and steering
inspection
![Page 21: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/21.jpg)
Bonferroni Method for Selecting the Best
• System with maximum expected performance is to be selected.
• System with maximum performance and maximum distance to the second best is to be selected.
i - max j i j
![Page 22: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/22.jpg)
Bonferroni Method for Selecting the Best -contd
1.- Specify , and R0
2.- Make R0 replications for each of the K systems
3.- For each system i calculate Yi*
4.- For each pair of systems i and j calculate Sij2 and select the
largest Smax2
5.- Calculate R = max{R0, t2 Smax2 / 2}
6.- Make R-R0 additional replications for each of the K systems
7.- Calculate overall means Yi** = (1/R) Yri
8.-Select system with largest Yi** as the best
![Page 23: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/23.jpg)
Statistical Models to Estimate the Effect of Design Alternatives
• Statistical Design of Experiments– Set of principles to evaluate and maximize the
information gained from an experiment.
• Factors (Qualitative and Quantitative), Levels and Treatments
• Decision or Policy Variables.
![Page 24: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/24.jpg)
Single Factor, Randomized Designs
• Single Factor Experiment– Single decision factor D ( k levels)– Response variable Y
– Effect of level j of factor D, j
• Completely Randomized Design– Different r.n. streams used for each replication
at any level and for all levels.
![Page 25: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/25.jpg)
Single Factor, Randomized Designs -contd
• Statistical model
Yrj = + j + rj
whereYrj = observation r for level j
= mean overall effect
j = effect due to level j
rj = random variation in observation r at level j
Rj= number of observations for level j
![Page 26: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/26.jpg)
Single Factor, Randomized Designs -contd
• Fixed effects model– levels of factors fixed by analyst rj normally distributed
– Null hypothesis H0: j = 0 for all j=1,2,..,k
– Statistical test: ANOVA (F-statistic)
• Random effects model– levels chosen at random j normally distributed
![Page 27: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/27.jpg)
ANOVA Test
• Levels-replications matrix
• Compute level means (over replications) Y.i* and grand mean Y..*
• Variation of the response w.r.t. Y..*
Yrj - Y..* = (Y.j* - Y..*) + (Yrj - Y.j*)
• Squaring and summing over all r and j
SSTOT = SSTREAT + SSE
![Page 28: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/28.jpg)
ANOVA Test -contd• Mean square MSE = SSE/(R-k) is unbiased estimator of
var(Y). I.e. E(MSE) = 2
• Mean square MSTREAT = SSTREAT/(k-1) is also unbiased estimator of var(Y).
• Test statistic
F = MSTREAT / MSE
• If H0 is true F has an F distribution with k-1 and R-k d.o.f.
• Find critical value of the statistic F1-
• Reject H0 if F > F1-
![Page 29: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/29.jpg)
Metamodeling
• Independent (design) variables xi, i=1,2,..,k
• Output response (random) variable Y
• Metamodel– A simplified approximation to the actual
relationship between the xi and Y
– Regression analysis (least squares)– Normal equations
![Page 30: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/30.jpg)
Linear Regression
• One independent variable x and one dependent variable Y
• For a linear relationship
E(Y:x) = 0 + 1 x
• Simple Linear Regression Model
Y = 0 + 1 x +
![Page 31: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/31.jpg)
Linear Regression -contd• Observations (data points)
(xi,Yi) i=1,2,..,n
• Sum of squares of the deviations i2
L = i2 = [ Yi - 0
’ - 1(xi - x*)]2
• Minimizing w.r.t 0’ and 1 find
0’* = Yi /n
1*
= Yi (xi - x*)/ (xi - x*)2
0* = 0
’* - 1*
x*
![Page 32: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/32.jpg)
Significance Testing• Null Hypothesis H0: 1 = 0
• Statistic (n-2 d.o.f)
t0 = 1*
/(MSE/Sxx)
where
MSE = (Yi - Ypi)/(n-2)
Sxx = xi2 - ( xi )2/n
• H0 is rejected if |t0| > t/2,n-2
![Page 33: Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs](https://reader030.vdocuments.mx/reader030/viewer/2022032704/56649d4d5503460f94a2b72b/html5/thumbnails/33.jpg)
Multiple Regression
• Models
Y = 0 + 1 x1 + 2 x2 + ... + m xm +
Y = 0 + 1 x + 2 x2 +
Y = 0 + 1 x1 + 2 x2 + 3 x1 x2 +