chapter 6 statistical analysis of output from terminating simulations
TRANSCRIPT
![Page 1: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/1.jpg)
Chapter 6
Statistical Analysis of Output from Terminating Simulations
![Page 2: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/2.jpg)
Chapter 5 – Detailed Modeling and Terminating Statistical Analysis
Simulation with Arena 2
Statistical Analysis of Output from Terminating Simulations
• Random input leads to random output (RIRO)• Run a simulation (once) — what does it mean?
– Was this run “typical” or not?– Variability from run to run (of the same model)?
• Need statistical analysis of output data– From a single model configuration– Compare two or more different configurations– Search for an optimal configuration
• Statistical analysis of output is often ignored– This is a big mistake – no idea of precision of results– Not hard or time-consuming to do this – it just takes a little planning
and thought, then some (cheap) computer time
![Page 3: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/3.jpg)
Chapter 5 – Detailed Modeling and Terminating Statistical Analysis
Simulation with Arena 3
Output Analysis
Output analysis is concerned with• Designing replications
Obtain most reliable info with minimum number of replications and minimum run length.
• Computing statisticsPoint and confidence interval estimationSize and independency issues
• Presenting them in a textual and graphical format.Aim is to understand the system behavior and
generate predictions for it!
![Page 4: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/4.jpg)
Chapter 5 – Detailed Modeling and Terminating Statistical Analysis
4Simulation with Arena
Time Frame of Simulations
• Terminating: Specific starting, stopping conditions– Run length will be well-defined (and finite)
• Steady-state: Long-run (technically forever)– Theoretically, initial conditions don’t matter (but
practically they usually do)– Not clear how to terminate a simulation run
• This is really a question of intent of the study• Has major impact on how output analysis is done• Sometimes it’s not clear which is appropriate
![Page 5: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/5.jpg)
Asli Sencer 5
Model 6.1• Same as Model 5.3
Number of trunk lines=26No additional staff during 5-8 hrs.
• 10 runs are made
For terminating case, make IID replications Run>Setup>Replication Parameters:
Number of Replications =10
Check both boxes for Initialize Between Replications• Outputs are saved to .dat files Statistics Module, Type=output, Data file name= Filename.dat
![Page 6: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/6.jpg)
Asli Sencer 6
Outputs of Model 6.1• Category Overview report will have some statistical-
analysis results of the output across the replications
![Page 7: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/7.jpg)
Asli Sencer 7
Output Precision in Model 6.1This information (except standard deviation) is in Category Overview report
If > 1 replication specified, Arena uses cross-replication data as aboveFor other confidence levels or graphics – Output Analyzer
![Page 8: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/8.jpg)
8
Confidence Interval EstimationNotation:number of replicationssample meansample standard deviationcritical value for significance level
(1-α)x100% Confidence Interval (CI): for for ,
![Page 9: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/9.jpg)
9Simulation with Arena Chapter 5 – Detailed Modeling and Terminating Statistical Analysis
Interpretation of a Confidence Interval
• A CI is an interval with random (data-dependent) endpoints that’s supposed to have stated probability (1-α) of containing, or covering, the expected values– “Target” expected value is a fixed, but unknown, number– Expected value = average of infinite number of replications
• A CI is not an interval that contains, say, 95% of the data generated by simulation outputs.– That’s a prediction interval … useful too, but different
• 95% CI means: If we make infinitely many batch runs of size and construct C.I. for α=5%, 95% of the time this interval will contain the true (but unknown) population parameter.
![Page 10: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/10.jpg)
10
Required Number of Replications to Achieve a Certain Precision
• Half-width =• We prefer confidence intervals with smaller half width — for
higher precision• Question: How many replications are needed to get the
required (better) precision? We can’t control t or s. We may just increase n — how much?
– Trial and error (now)– Approximate number for acceptable precision (below)– Sequential sampling (Chapter 11)
Want this to be “small,” say< h where h is prespecified
Chapter 5 – Detailed Modeling and Terminating Statistical AnalysisSimulation with Arena
![Page 11: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/11.jpg)
Chapter 5 – Detailed Modeling and Terminating Statistical Analysis
11Simulation with Arena
Half Width and Number of Replications
• Let required half-width, solve for• Not really solved for n (t, s depend on n)• Approximation: • Replace t by z, corresponding normal critical value
– Pretend that current s will hold for larger samples
– Get
• Easier but different approximation:
s = sample standarddeviation from “initial”number n0 of replications
h0 = half width from “initial” number n0 of replications
n grows quadraticallyas h decreases.
𝑛=𝑡❑2𝑛− 1,1−∝/2
𝑠2
h2
𝑛≅ 𝑧❑21−∝/2
𝑠2
h2
𝑛≅𝑛0h02
h2
![Page 12: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/12.jpg)
Asli Sencer 12
Number of Replications Needed
• If we require h=$250 rather than $812 for total cost,
replications are needed.
𝑛≅𝑛0h02
h2=10 (812.76 )2
(250 )2=105.7
![Page 13: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/13.jpg)
Asli Sencer 13
Model 6.2
• 110 Runs• 95% CI for the total cost= 22,175.19 +- 369.54, half width=369.54>250! Required accuracy is still not met, why? As more replications are made, in formula, does not change, increases; but , the sample standard deviation might increase. So might decrease.• 95% CI for the %rejected calls= 11.73.96 +- 0.51, accuracy is quite satisfactory!
![Page 14: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/14.jpg)
Asli Sencer 14
Model 6.3
• 1000 Runs-as a trial• Save the output to Total cost.dat• Open Output analyzer as a separate application File>Data File>Export Export binary data in .dat file to a plain ASCII text file and save.• Open Arena Input Analyzer Plot the histogram of the Total Costs
![Page 15: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/15.jpg)
Asli Sencer 15
Histogram of 1000 Total Cost Values
• Since Total Cost values is a sum, law of large numbers apply. We see that the distribution approaches normal as the number of replications increase!
• Same is true for average statistics due to central limit theorem.• It is not true for extreme value statistics like maximum or minimum.
![Page 16: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/16.jpg)
Chapter 5 – Detailed Modeling and Terminating Statistical Analysis
16Simulation with Arena
Confidence Intervals (cont’d)
• Usual formulas assume normally-distributed data Never true in simulation Might be approximately true if output is an
average, rather than an extreme Central limit theorem Issues of robustness, coverage, precision –
details in book
![Page 17: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/17.jpg)
Chapter 5 – Detailed Modeling and Terminating Statistical Analysis
17Simulation with Arena
Comparison of AlternativesStatistical Hypothesis Test
0
0
211
210
H
H
Reject Ho if is significantly large or small, i.e.,
performance of system 1 is significantly different than system 2!
Here: : Total Cost of Base Model (110 observations)
: Total Cost of Alternative Model (110 observations)
21 XX
1x
is the mean performance of system ii
2x
![Page 18: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/18.jpg)
Asli Sencer 18
Comparing Two Scenarios
• Base Scenario: Model 6.4 (Same as in Model 5.3) -110 runs -26 Trunk Lines, No New Staff between 12:00-16:00• Alternative scenario: Model 6.4 (More-resources scenario) -110 runs -29 Trunk Lines, (Change the capacity from 26 to 29) -Hire three for each of Larry, Moe, Curly, Hermann and Sales Resources. (Change these variables from 0 to 3)• Tradeoff is between increased salary cost but decreased excess
waiting costs. Will the total costs decrease?• Percent Rejected calls will decrease, but how much?
![Page 19: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/19.jpg)
Asli Sencer 19
Comparison of Scenarios• Runs both models for 110 times.• Statistics Data Module Save output files –BaseCase.dat or -MoreResources.dat.• 95% CI for total costs are Base model: 22,175.19 +- 369.54=[21,805, 22,544] Increased resources: 24,542.82 +- 329.11=[24,213, 24,871] Intervals do NOT overlap, hence Total Costs are significantly different at 5% significance level. • 95% CI for percent rejected are Base model: 11.74 +- 0.51=[11.23, 12.25] Increased resources: 1.73 +- 0.31=[1.42, 2.04] Intervals do NOT overlap, hence Percent Rejected are significantly different at 5% significance level.
![Page 20: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/20.jpg)
Asli Sencer 20
Arena Output Analyzer
• Separate application in Arena• Operates in output files (.dat) generated by
Arena through the Statistics data module• Data in .dat file is in binary format to be opened
by Arena Output Analyzer only!• Provides confidence intervals on expected output
statistics as also appear in Arena output reports.• Provides statistical comparison of two scenarios,
and others.
![Page 21: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/21.jpg)
Asli Sencer 21
Comparison of Scenarios with Arena Output Analyzer
• Open Output Analyzer• Select File>New to open a data group, i.e., list of .dat files• Add TotalCost-BaseCase.dat TotalCost-MoreResources.dat PercentRejected-BaseCase.dat PercentRejected-MoreResources.dat• Can save this data group as .dgr file to refer easily
afterwards• Analyze>Compare Means Add each pair of comparisons by choosing ‘lumped’ so that all 110 values are considered in the analysis
![Page 22: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/22.jpg)
Asli Sencer 22
Hypothesis Tests
• Ho: Mean TC of base case = Mean TC of more resources Ha: Mean TC of base case ≠ Mean TC of more resources
• Ho: Mean % rejected of base case = Mean % rejected of more resources case
Ha: Mean % rejected of base case ≠ Mean % rejected of more resources case
![Page 23: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/23.jpg)
Asli Sencer 23
Output Report-Compare Means
Confidence interval on difference misses 0, so conclude that there is a (statistically) significant difference between the base model and the alternative at α=5%
![Page 24: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/24.jpg)
Asli Sencer 24
Evaluating Many Scenarios with Process Analyzer
• Separate application in Arena• Allows making multiple pairwise scenario comparisons
at a time.• PAN operates on Arena program files with .p
extension, generated when .doe model is run.• A PAN scenario includes a program file, a set of values
for the input controls (decision variables in the form of variables and resources), a set of output responses.
• A PAN project is a collection of such scenarios that can be saved by .pan extension for future reference.
![Page 25: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/25.jpg)
Asli Sencer 25
Development of a PAN Project• Use Model 6.5 110 runs Output data files are deleted since they will be useless in PAN.• Open a PAN project File > New, File > Open • Add a new scenario. Double click on the raw Name=Base Case, Program File=Model 6.5.p Replications=110• Add contols Right click in this line OR Insert > Control Under Resources: The capacity of trunk line Under User Specified: New Tech 1, New Tech 2, New Tech 3, New Tech All, New Sales• Add responses Right click on this line OR Insert > Responses Under user specified: Total Cost, Percent Rejected
![Page 26: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/26.jpg)
26
New Scenarios• Suppose you have $1360/week to spend on all additional
resources. To which of the six expandable resources should you allocate the new money?
• Then following 6 alternative scenarios apply in addition to Base Case.
13 more trunk lines ($98 each) 4 more tech 1,2,3 people ($320 each) 3 more tech all people ($360 each) 4 more sales people ( 340 each)• Run the scenarios Check the scenarios to run Run > Go OR play button OR F5 function key
Asli Sencer
![Page 27: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/27.jpg)
Asli Sencer 27
PAN screen
![Page 28: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/28.jpg)
Asli Sencer 28
Generating Reports for Multiple Comparison in PAN
• Insert > Chart OR right click on a response column. Chart type=Box whiskers Check Identify Best Scenarios box Select ‘smaller is the better’• Red boxes are significantly better than blue ones at 5% significance
level.• To decrease the half width of a scenario, increase the number of
replications of that specific one.• Error tolerance is a positive value that represents an amount small
enough that you don’t care if the selected scenarios are actually inferior to the true best one by at most this amount.
A positive error reduce the number of selected scenarios at the risk of being off by a little bit.
![Page 29: Chapter 6 Statistical Analysis of Output from Terminating Simulations](https://reader035.vdocuments.mx/reader035/viewer/2022081503/56649e205503460f94b0c479/html5/thumbnails/29.jpg)
Asli Sencer 29
A PAN Report