simulation tactics, strategy and interpretation pdfauthor
TRANSCRIPT
Simulation Tactics, Strategy and Interpretation
G A Vignaux
August 29, 2004
1
Contents
1 Introduction 3
2 Developing Simulations 5
3 Traces 9
4 Terminating versus non-terminating simulations 15
5 Output Analysis 23
6 Variance reduction Methods 29
7 Other Simulation Topics 36
2
1 Introduction
• A simulation must be written to answer a question, not as ageneral toy.
• Ask what are the (specific) effects of doing something. Forexample: “What is the change in average delay time if weincrease the number of servers in the system by 1?”
• You must interpret the output with care. Consider the risksinvolved in a wrong decision.
• A simulation program must be verified (does it correctly replicatethe designed model?) and validated (does it correctly replicatethe real situation?)
3
1.1 Statistical nature of simulation
• Runs of the simulation are statistical experiments and must beanalysed as such.
• We must be able to distinguish between (1) variations due tochanges in the system configuration and (2) sampling variation inthe results.
• Serial correlation in observations give us problems
• Control over random variables is a great help.
4
2 Developing Simulations
Design, Language, developing code, testing, validation.
2.1 Design
Initial Design of the model. What questions are to be answered?Modelling with ACDs (2.1.1).
5
2.1.1 ACD
6
2.2 Simulation Language
Choice of a simulation languages. Advantages and limitations ofdifferent approaches.
Compiled: Simula, Simscript, C-sim, C++-sim
Scripted: SimPy
Packages: SIMON, HOCUS,
Graphical: Simul8
7
2.3 Developing code
Develop in an incremental manner. DO NOT write all the code outat once.
Use Deterministic times at first, NOT random ones.
2.4 Testing
Testing, Traces (see Section 3).
Unit tests.
How do you test a system with random numbers? If you have controlover RVs you can find out what they should be.
What is the difference between validation and verification?
8
3 Traces
Trace files are essential for developing, debugging (Section 3.2), andrunning a simulation.
• It is a text file generated by the simulation.
• Consists of a series of lines. Each line records an event.
• Events: instants when the state of the model changes
• A trace line consists of:
– simulation time always first
– Followed by any other information about the event
• This format is chosen so some, now unknown, program can readand analyse the trace later. (eg S-Plus, R, )
9
Example 3.1
10.0325 Customer 3 arrives
11.0567 Customer 2 departs
11.0567 Customer 2 leaves the system
11.0567 Customer 3 starts service at node 5
12.0033 Customer 3 departs
12.0033 Customer 3 transfers to node 4
The first element on a line is the time. It is possible to work out when
customer 3 arrived, went into service, and departed.
10
3.1 Producing a trace
Either by print statements or by aspecial version of the simulationpackage.
The simplest way of producing a trace is to include conditional tracestatements in the code at suitable points. These can be printstatements or calls to a trace method.
11
Example 3.2 Here is an example of a trace method for a SimPy
Customer process:
def trace(self,message):
if tracing:
print ’%7.4f pax %03d %s’%(now(), self.id, message)
If the boolean tracing was set to True (or 1), a statement like
self.trace(’arrived’)
would print a trace line with the simulation time, the customer’s
identification number, and the message arrived.
12
3.2 Traces for debugging
Simulation programs are notoriously difficult to get running properly.They involve running artificial parallel interacting processes - ratherlike simple operating systems. The particular complication is therandom numbers built into most simulations.
We always build-in the ability to generate a trace file while runningthe program. We should be able to switch it on or off by setting aparameter for the run so that the trace does not interfere with theresults of the run.
Using this technique it is usually easy to see any flaws in the logic ofthe model. If, as we should, we start by using non-random variableswhen we develop the model this logical verification is much easierwith a trace.
13
3.3 Traces for output
Traces are often use as an intermediate file between the simulationand analysis of the output. The trace records a complete run of themodel. It it is written so it is easy for another program to read in,much of the statistical analysis and presentation can be carried outby programs designed to do just that. For example the trace outputof a Simscript model might be read in by an S-PLUS script to findserial correlations.
It may be useful to run the trace through another script (in Perl,Python or shell script) to massage the data before sending it to thestatistical package.
14
4 Terminating versus non-terminating
simulations
There are two general classes of system simulation models:
4.1 Terminating
Some models terminate naturally, such as a shop closing its doors atthe end of the day. To get more accuracy runs have to be replicated.
In many cases we will be interested in transient measurements: suchas those where we want to measure the starting effects. For example,the performance of the airport system when a 747 arrives atAuckland.
15
4.2 Non-Terminating
Non-terminating Some models have no natural termination, such as a24-hr bus terminal. To get more accuracy, runs have to be increasedin length.
We would probably be interested in making steady statemeasurements: such as those needed to compare with theoreticalcalculations in queue or queue network theory. This leads to sometricky decisions in running the simulations.
16
4.2.1 Achieving steady state for non-terminatingsimulations
0 2 4 6 8 10 12 14 16 18 20
t
Number of restaurants
0
5
10
15
20
25
30
35
40y
For non-terminating simulations we want to measure operations in“steady-state”.
17
The easiest way to start a simulation is to have no activity occurringand all queues empty. (“Empty and Idle”). But as this state is nottypical, this can bias the results.
Two ways out of this difficulty are to use a run-in( or warm-up)period or to start at a typical state.
18
4.2.2 Using a warm-up period
0 2 4 6 8 10 12 14 16 18 20
t
Price of Pizza
0
2
4
6
8
10
12
14
16
18
20y
Reject the results for the warm-up period. The simulation properbegins at the end of the warm-up. Statistics are collected from thenon.
19
4.2.3 How long should the warm-up period be?
There seems to be no sure statistical test for this. One way is tomonitor statistics through an entire run and check for the time steadystate is reached. Nowadays this can be done using graphical outputto check, for example, that average queue lengths are not graduallyincreasing any more. Otherwise graphs of trace output may suffice.
Another is to determining the run-in period in advance and use thesame run-in period for all the experiments. This run-in period maybe determined by hunch, from an initial simulation or by analysis ofa simplified model.
20
4.2.4 Starting in a typical state
0 20 40 60 80 100
t
Number of restaurants
10
15
20
25
30
35
40y
An alternative is to begin each run of the simulation withnon-empty conditions that are somehow typical.
21
These conditions may be quite complicated. They might bedetermined from knowledge of the system, from an analysis of theACD, or from the results of preliminary simulations.
This may be difficult to set up; e.g. if you spread entities round thesystem, all waiting to start the next activity how do you get responsetimes for those initial ones?
22
5 Output Analysis
• Simple statistics (in the Simulation language). SimPy Monitors.
• Graphical Output (5.1)
• Simple Comparison of Runs
• Statistical Analysis of Results (5.2)
23
5.1 Graphical Output
Simple types of output, eg graphics. Graphs, Histograms.
24 26 28 30 32 34
x
Number of restaurants
0
50
100
150
200
250
300
350
400
450
500
550
600y
Example.
24
5.2 Statistical Analysis
Estimation is simplest when the observations are statisticallyindependent and identically distributed. Then the central limittheorem, implying Normal distributions for totals and hence means,can be used to estimate ranges of uncertainty or confidence intervals.
But in many simulations, observations of a single variable are likelyto be autocorrelated. Then, even though the estimates are not biasedour estimates of uncertainty are more difficult.
• Measure the Correlation and correct for it
• Replication
• Batching
• Regeneration
25
5.2.1 Simple Replication
The simulation is repeated several times with the same conditionsexcept that different streams of random number are used for eachrun. This ensures that the replications are independent of each other.Thus for a random variable X resulting from n replications of thesame simulation, the mean value is the overall mean of the n
replications and the overall variance is the mean of the n variances.
26
5.2.2 Batching
A single long run is divided into batches of a given number ofcompletions or a fixed time. The measurements made in the batcheswill be correlated but not very much if the batches are long. Thebatches are then regarded as replications.
27
5.2.3 Regeneration
A single long simulation run is broken into batches by dividing therun at regeneration points. These are identical system states wherethe system appears to start anew in a statistical sense. Thus anM/M/1 queue would have a regeneration point whenever a job leavesit empty.
Although the batches are not the same length, the methodguarantees that there is no correlation between them. The analysis isdescribed in [1, Ch 23.4].
28
6 Variance reduction Methods
These are methods to increase the precision of the response variableestimates. They are often called Monte Carlo methods. They wereinvented primarily for use in statistical methods of estimatingintegrals and can be particularly effective there.
Results in simulating real discrete-event systems are sometimesdisappointing. See [1, Ch 23].
Some methods include (6.1) Common Random Numbers, (6.2)Control Variates, (6.3) Antithetic Variates, and ((6.4) ImportanceSampling.
29
6.1 Common random numbers
This takes advantage of the control we have over the streasm ofrandom numbers used in the simulation.
One can hold the sampling variation constant across the alternativepolicies by using the same random numbers in each run. Thedifferences between results is then (mainly) due to the differentpolicies.
Each source of variation in the model can have its own stream ofrandom numbers. For example, the random variates used to generateinter-arrival times, the CPU service time, and the choice of pathswould all be from different independent streams.
30
In SimPy, this would be done using different random variable objectsfor each source of variation. For different runs the seeds for eachstream are reset to the same initial values. Then, if synchronisationof the random numbers is achieved, each entity retains the samevalues of random variables for corresponding actions in all the runsand behaves in a similar way. We hope that sampling variation willbe minimised.
There is positive correlation between runs. The variance of thedifferences between policies is reduced by the covariance between thecorresponding runs.
31
6.2 Control Variates
Additional observations are made of variables that might be expectedto correlate with those being measured.
For example, if we are interested in the average response times in aqueue and measure W , we might also observe the average servicetimes we generate, S. If a service time is high we would expect thecorresponding response time to be high. We also know what the truemean of these service times, Ts, should be (since we are generatingthem from the service time distribution).
32
We correct the observed average response time by (some proportionof) the difference between the actual and expected service timemeans,
W∗
= W − λ(S − Ts),
where W∗
is the corrected mean response time. The value of λ canbe established by regression measurements in pilot simulation runs.Note that no matter what value of λ is chosen, the estimate of W
∗
will remain unbiassed but its variance may be increased if you choosebadly.
33
6.3 Antithetic Variates
Each run is duplicated. The second run in a pair uses an identicalrandom number stream to the first BUT corresponding RVs arenegatively correlated. This is usually done by replacing each PRN Ui
(uniform 0 to 1) by its antithetic value 1− Ui. Then a large servicetime in the first run will have a correspondingly small service time inthe antithetic run.
Thus the two runs should have resulting measurements X1 and X2
that are negatively correlated to some degree. We use the mean ofthese, (X1 + X2)/2 as the resulting measurement. This should haveless sampling variance than either of the two or even of a run thesame length of the two put together.
34
6.4 Importance Sampling
If we are interested in system states that occur very rarely (eg rarefailures) we may distort the probabilities so as to increase the chanceof this happening in our simulation. We then adjust the results forthis distortion. This requires a relatively simple situation so we canproduce a probability model so the effect of the distortion can beallowed for..
Dr Peter Smith (ex-MCS, now at Canterbury) is using this techniqueto study the effect of error detection and correction methods inTelecom transmissions. In these studies bit error rates of 1 in 1011
are typical. There is no way to simulate this without the leverage ofimportance sampling.
35
7 Other Simulation Topics
• Stochastic Optimisation Methods
• Genetic Algorithms
• Simulated Annealing
• Gibbs Sampling
36
References
[1] Frederick S Hillier and Gerald J. Lieberman. Introduction toOperations Research. McGraw-Hill, 5th edition, 1990.
[2] Averill M Law and W D Kelton. Simulation Modeling andAnalysis. McGraw-Hill, 1982.
[3] Michael Pidd. Computer simulation in management science.John Wiley & sons Ltd, Chichester, 2nd edition, 1988.
[4] Michael Pidd, editor. Computer Modelling for discrete simulation.Wiley, 1989.
[5] Wayne L Winston. Operations Research, Applications andAlgorithms. Duxbury Press, 4th edition, 2004.
37