8/10/2019 SSM-UNIT 7 8

1/17

1

VERIFICATION AND VALIDATION OF SIMULATIONMODELS

VERIFICATION: is concerned with buildin the !odel correctl"# It $roceedes

b" co!$%rison o& the conce$tu%l !odel to the co!uter re$resent%tion th%t

i!$le!ents th%t conce$tion#

VALIDATION: is concerned with buildin the correct !odel# It %tte!$ts to

con'r! th%t % !odel is %n %ccur%te re$resent%tion o& the re%l s"ste!# It is

%chie(ed throuh the c%liber%tion o& the !odel#

GOAL OF THE VALIDATION PROCESS:

To $roduce % !odel th%t re$resents true s"ste! beh%(iour closel" enouh

&or the !odel to be used %s % substitute &or the %ctu%l

s"ste! &or the $ur$ose o& e)$eri!entin with thes"ste!* %n%l"+in s"ste! beh%(iour %nd

$redictin s"ste! $er&or!%nce#

To incre%se to %n %cce$t%ble le(el the credibilit" o&

the !odel* so th%t the !odel will be used b"

!%n%ers %nd other decision !%,ers#

MODEL BUILDING, VERIFICATION AND VALIDATION

Model buildin-

o

First ste$ consists o& obser(in there%l s"ste! %nd the inter%ction

%!on their (%rious co!$onents

%nd o& collectin d%t% on their

beh%(iour# .ersons &%!ili%r with

the s"ste! or %n" subs"ste! should be /uestioned to %in

s$eci%l ,nowlede# O$er%tors* technici%ns* re$%ir %nd

!%inten%nce $ersonnel* enineers* su$er(isors %nd !%n%ers

underst%nd so!e %s$ects o& s"ste! th%t !iht be un&%!ili%r to

others#

o Second ste$ is the construction o& % conce$tu%l !odel0 %

collection o& %ssu!$tions %bout the co!$onents %nd the

structure o& the s"ste!* $lus h"$othesis %bout the (%lues o&

!odel in$ut $%r%!eters#

o Third ste$ is i!$le!ent%tion o& %n o$er%tion%l !odel*usu%ll" b"

usin si!ul%tion so&tw%re %nd incor$or%tin the %ssu!$tions o&

8/10/2019 SSM-UNIT 7 8

2/17

the conce$tu%l !odel into the world(iew %nd conce$ts o& the

si!ul%tion so&tw%re#

The !odel builder will return to e%ch o& these ste$s !%n" ti!es while buildin*

(eri&"in %nd (%lid%tin the !odel#

VERIFICATION OF SIMULATION MODELS

The $ur$ose o& !odel (eri'c%tion is to %ssure th%t the conce$tu%l !odel is

re2ected %ccur%tel" in the o$er%tion%l !odel# The conce$tu%l !odel /uite

o&ten in(ol(es so!e deree o& %bstr%ction %bout s"ste! o$er%tions or so!e

%!ount o& si!$li'c%tion o& %ctu%l o$er%tions#

So!e suestions in the (eri'c%tion $rocess-

3%(e the o$er%tion%l !odel chec,ed b" so!eone other th%n its de(elo$er*

$re&er%bl" %n e)$ert in the si!ul%tion so&tw%re bein used#

M%,e % 2ow di%r%! th%t includes e%ch loic%ll" $ossible %ction % s"ste! c%n

t%,e when %n e(ent occurs* %nd &ollow the !odel loic &or e%ch %ction &or e%ch

e(ent t"$e#

3%(e the i!$le!ented !odel dis$l%" % wide (%riet" o& out$ut st%tistics

%nd e)$l%in the! %ll#

3%(e the o$er%tion%l !odel $rint the in$ut $%r%!eters %t the end o& the

si!ul%tion* to be sure th%t these $%r%!eter (%lues h%(e not been ch%ned

in%d(ertentl"#

M%,e the o$er%tion%l !odel %s sel&0docu!entin %s $ossible#

I& the o$er%tion%l !odel is %ni!%ted* (eri&" th%t wh%t is seen in the %ni!%tion

i!it%tes the %ctu%l s"ste!#

The Inter%cti(e Run Controller 4IRC5 or debuer is %n essenti%l co!$onent o&

success&ul si!ul%tion !odel buildin# The IRC %ssists in 'ndin %nd correctin

those errors in the &ollowin w%"s-

o The si!ul%tion c%n be !onitored %s it $roresses#

o Attention c%n be &ocused on % $%rticul%r entit" line*o& code* or

$rocedure#

o V%lues o& selected !odel co!$onents c%n be obser(ed#

o The si!ul%tion c%n be te!$or%ril" sus$ended* or $%used*

not onl" to (iew in&or!%tion* but %lso to re%ssin (%lues orredirect entities#

6r%$hic%l inter&%ces

%rereco!!ended &or

%cco!$lishin(eri'c%tion

%nd (%lid%tion#

8/10/2019 SSM-UNIT 7 8

3/17

7

PROBLEM 1:

8hen (eri&"in the o$er%tion%l

!odel 4in % ener%l $ur$ose l%nu%e such %s FORTRAN* .%sc%l* C or C99*

or !ost si!ul%tion l%nu%es5 o& the sinle0ser(er /ueue !odel %sin '

below-

An %n%l"st !%de % run o(er 1: units o& ti!e %nd obser(ed th%t the ti!e0co(er%e lenth o& the w%itin line w%s L; custo!er#

The t%ble below i(es the h"$othetic%l $rintout &ro! si!ul%tion ti!e

CLOC to CLOC STATUS=>

CLOC

CLOC5 9 41>54759 4>>541159 41>541115941541:15

47>5947594115941115941:15

= B1: = >#G7

8/10/2019 SSM-UNIT 7 8

4/17

G

CALIBERATION AND VALIDATION OF MODELS

Veri'c%tion %nd (%lid%tion* %lthouh conce$tu%ll" distinct* usu%ll" %re

conducted si!ult%neousl" b" the !odeler#

V%lid%tion is the o(er%ll $rocess o&

co!$%rin the !odel %nd its beh%(iour

to the re%l s"ste! %nd its beh%(iour#

C%liber%tion is the iter%ti(e $rocess o&

co!$%rin the !odel to the re%l s"ste!*

!%,in %ddition%l %d@ust!ents 4or e(en

!%@or ch%nes5 to the !odel* co!$%rin

the re(ised !odel to re%lit"* !%,in

%ddition%l %d@ust!ents* co!$%rin

%%in* %nd so on#

Fi below shows the rel%tionshi$ o& !odelc%liber%tion to the o(er%ll (%lid%tion $rocess#

The co!$%rison o& the !odel to re%lit"

is c%rried out b" % (%riet" o& tests#

so!esub@ecti(e %nd others ob@ecti(e#

o Sub@ecti(e - usu%ll" in(ol(e $eo$le* who %re ,nowlede%ble %bout

one or !ode %s$ects o& the s"ste!* !%,in @ude!ents %bout the

!odel %nd its out$ut#

o Ob$ective % re&uire !ata on t'e systems be'aviour( aso corres"on!in) !ata

$roduced b" the !odel#o Then one or !ore st%tistic%l tests %re $er&or!ed to co!$%re so!e

%s$ect o& thes"ste! d%t% set with the s%!e %s$ect o& the !odel d%t%

set#

This iter%ti(e $rocess o& co!$%rin !odel with s"ste! %nd then re(isin

both the conce$tu%l %nd o$er%tion%l !odels to %cco!!od%te %n"

$ercei(ed !odel de'ciencies is continued until the !odel is @uded to be

suHcientl" %ccur%te#

N%"lor %nd Finer &or!ul%ted % three0ste$ %$$ro%ch th%t h%s been widel"

&ollowed#

o uild % !odel th%t h%s hih &%ce (%lidit"#

o V%lid%te !odel %ssu!$tions

o Co!$%re the !odel in$ut0out$ut tr%ns&or!%tions to corres$ondin

in$ut0out$ut tr%ns&or!%tions &or the re%l s"ste!#

Face validit:

8/10/2019 SSM-UNIT 7 8

5/17

o The 'rst o%l o& the si!ul%tion !odeler is to construct % !odel th%t

%$$e%rs re%son%ble on its &%ce to !odel users %nd others who %re

,nowlede%ble %bout the re%l s"ste! bein si!ul%ted#

o .otenti%l users %nd ,nowlede%ble $ersons c%n %lso e(%lu%te !odel

out$ut &or re%son%bleness %nd c%n %id in identi&"in !odel

de'ciencies#

o Sensitivity anaysis can aso be use! to c'ec* a mo!es face vai!ity+ T'e mo!e

user is %s,ed whether the !odel beh%(es in the e)$ected w%" when

one or !ore in$ut (%ri%bles %re ch%ned#

Validati!" !# $!del a%%&$'ti!"%:

o Model %ssu!$tions &%ll into two ener%l cl%sses-

Structur%l %ssu!$tions- in(ol(e /uestions o& how the s"ste!

o$er%tes %nd usu%ll" in(ol(e si!$li'c%tions %nd %bstr%ctions

o& re%lit"#

D%t% %ssu!$tions- should be b%sed on the collection o& reli%bled%t% %nd

correct st%tistic%l %n%l"sis o& the d%t%#

o The use o& oodness0o&0't tests is %n i!$ort%nt $%rt o& the

(%lid%tion o& d%t% %ssu!$tions#

Validati"( i"'&t)!&t'&t t*a"%#!*$ati!"%:

o The ulti!%te test o& % !odel* %nd in &%ct the onl" ob@ecti(e test o& the

!odel %s % ,'oe( is t'e mo!es abiity to "re!ict t'e future be'aviour of t'e

rea system when the !odel in$ut d%t% !%tch the re%l in$uts %nd when

% $olic" i!$le!ented %t so!e $oint in the s"ste!#o The structure o& the !odel !ust be %ccur%te enouh &or the !odel to

!%,e

ood $redictions* not @ust &or one in$ut d%t% set* but &or the r%ne o&

in$ut d%t% sets th%t %re o& interest#

o The !odeler c%n %lso use historic%l d%t% th%t h%(e been reser(ed &or

(%lid%tion

$ur$oses onl"#

o In %n" c%se* the !odeler should use the !%in res$onses o&

interest %s the $ri!%r" criteri% &or (%lid%tin % !odel#

o The !odel will be used to co!$%re %ltern%ti(e s"ste! desins or toin(esti%te s"ste! beh%(iour under % r%ne o& new in$ut conditions#

o First* the res$onses o& the two !odels under si!il%r in$ut

conditions will be used %s the criteri% &or co!$%rison o& the

e)istin s"ste! to the $ro$osed s"ste!#

o Second* in !%n" c%ses the $ro$osed s"ste! is % !odi'c%tion o&

8/10/2019 SSM-UNIT 7 8

6/17

:

the e)istin s"ste! %nd the !odeler ho$es th%t con'dence in the

!odel o& the e)istin s"ste! c%n be tr%ns&erred to the !odel o&

the new s"ste!#

I"'&t)!&t'&t validati!": U%i"( +i%t!*ical i"'&t data:

o 8hen usin %rti'ci%ll" ener%ted d%t% %s in$ut d%t%* the !odeler

e)$ects the !odel to $roduce e(ent $%tterns th%t %re co!$%tible

with but not identic%l to* th%t e(ent $%tterns th%t occurred in the re%l

s"ste! durin the $eriod o& d%t% collection#

o An %ltern%ti(e to ener%tin in$ut d%t% is to use the %ctu%l historic%l

record* to dri(e the si!ul%tion !odel %nd then to co!$%re !odel

out$ut with s"ste! d%t%#

o E(ent schedulin without r%ndo! nu!ber ener%tion could

be i!$le!ented /uiet e%sil" in % ener%l $ur$ose

$ror%!!in l%nu%e or !ost si!ul%tion l%nu%es b"

usin %rr%"s to store the d%t% or re%din the d%t% &ro! the'le#

o 8hen usin the %bo(e techni/ue* the !odeler ho$es th%t the

si!ul%tion will du$lic%te %s closel" %s $ossible the i!$ort%nt

e(ents th%t occurred in the re%l s"ste!#

o In so!e s"ste!s* electronic counters %nd de(ices %re used

to e%se the d%t% collection t%s, b" %uto!%tic%ll" recordin

cert%in t"$es o& d%t%#

I"'&t)!&t'&t validati!": U%i"( a t&*i"( te%t:

o In %ddition to st%tistic%l tests* or when no st%tistic%l test is

re%dil" %$$lic%ble* $ersons ,nowlede%ble %bout s"ste!

beh%(iour c%n be used to co!$%re !odel out$ut to s"ste!

out$ut#

o S"ste! $er&or!%nce re$orts %re r%ndo!l" shuJed %nd i(en

to the enineer* who is %s,ed to decide which re$orts %re &%,e

%nd which %re re%l#

o I& the enineer c%nnot distinuish between &%,e %nd re%l

re$orts with %n" consistenc"* the !odeler will conclude

th%t the

test $ro(ides no e(idence o& !odel in%de/u%c"#

PROBLEM :

T'e "ro!uction ine at t'e S,eet Li T'in)s Can!y -actory

in Dec%tur consists o& three !%chines th%t !%,e

$%c,%e* %nd bo) their &%!ous c%nd"# One

8/10/2019 SSM-UNIT 7 8

7/17

!%chine 4the c%nd" !%,er5 !%,es %nd wr%$s

indi(idu%l $ieces o& c%nd" %nd sends the! b"

con(e"or to the $%c,er# The second !%chine 4the

$%c,er5 $%c,s the indi(idu%l $ieces into % bo)# A

third !%chine 4the bo) !%,er5 &or!s the bo)es

%nd su$$lies the! b" con(e"or to the $%c,er# The

s"ste! is illustr%ted %s in '#

V%lid%tion o& the C%nd"0F%ctor" Model-

Re%'!"%e, i S%te$, -i M!del, .i

1/ P*!d&cti!" level 02/30 004,153

/ N&$6e* !#!'e*at!*

4 4

4/ Ti$e !# !cc&*a"ce 2:, 0:71, 13:13 2:7, 0:7, 13:17

I"'&t data

%et 8

S%te$'*!d&cti

!",

M!del'*!d&cti!", 9i8

Obser(eddiKerence*

S/u%redDe(i%tion

&ro!Me%n*

8/10/2019 SSM-UNIT 7 8

8/17

1

1 02/30 004/153 1G*> #G)1>

/1 43/553 0 G#>)1>

4 245/ 271/73 0 1#77>)1>

7 22/4 200/43 *>77 1#7:)1>

5 05/743 017/13 11*G> 7#G)1>

.=0(121+3 =

1 In the %bo(e t%ble* Sd=

=.

SdB5B4>#B5

=1#7

I"'&t)O&t'&t Validati!" U%i"( a T&*i"( Te%t

In %ddition to st%tistic%l tests* or when no st%tistic%l test is re%dil"

%$$lic%ble* $ersons ,nowlede%ble %bout s"ste! beh%(iour c%n be

used to co!$%re !odel out$ut to s"ste! out$ut#

E- Su$$ose th%t re$orts o& s"ste! $er&or!%nce o(er diKerent

d%"s %re $re$%red* %nd si!ul%tion out$ut d%t% to $roduce '(e &%,e

re$orts# The 1> re$orts !ust be e)%ctl" in the s%!e &or!%t %nd %re

r%ndo!l" shuJed %nd i(en to the enineer* who is %s,ed to decidewhich is &%,e %nd which is re%l# I& the enineer identi'es %n" &%,e

re$orts* the !odel builder /uestions the enineer %nd uses the

in&or!%tion %thered to i!$ro(e the !odel# I& the enineer c%nnot

distinuish between re%l %nd &%,e re$orts* then !odeler will conclude

th%t the test $ro(ides no e(idence o& !odel in%de/u%c"#

OPTIMI-ATION VIA SIMULATION

E;a$'le%:

1/ Mate*ial% Ha"dli"( S%te$

8/10/2019 SSM-UNIT 7 8

9/17

dis$%tch the A6Vs# Altern%te desins will be e(%lu%ted

8/10/2019 SSM-UNIT 7 8

10/17

%ccordin to A6V utili+%tion* tr%ns$ort%tion del%" &or !%teri%l th%t

needs to be !o(ed* %nd o(er%ll in(est!ent %nd o$er%tion costs#

/ Li>&i?ed Nat&*al Ga%

8/10/2019 SSM-UNIT 7 8

11/17

!e!ber o& A6Vs* the lo%d $er A6V* %nd the routin %lorith! used to

dis$%tch the A6Vs res$ecti(el"#

8/10/2019 SSM-UNIT 7 8

12/17

8/10/2019 SSM-UNIT 7 8

13/17

%lorith!s th%t u%r%ntee con(erence to the lob%l o$ti!%lsolution %s the si!ul%tion eKort beco!es in'nite# Theseu%r%ntees %re use&ul bec%use the" indic%te th%t the %lorith!tends to et to where the %n%l"st w%nts it to o#

8/10/2019 SSM-UNIT 7 8

14/17

o O'ti$al #!* dete*$i"i%tic c!&"te*'a*t:The ide% here is to

use %n %lorith! th%t would 'nd the o$ti!%l solution i& the$er&or!%nce o& e%ch desin could be e(%lu%ted with cert%init"#

o R!6&%t +e&*i%tic%: !%n" heuristics h%(e been de(elo$ed &or

deter!inistic o$ti!i+%tion $roble!s th%t do not u%r%ntee

'ndin the o$ti!%l solutions* but ne(ertheless been shown tobe (er" eKecti(e on diHcult* $r%ctic%l $roble!s#

USING ROBUST HEURISTICS

The $rocedure does not de$end on stron $roble! structure0 such %s

continuit" or con(e)it" o& E4?4)1*)(8++(9!55* to be eKecti(e* c%n be %$$lied

to $roble!s with !i)ed t"$e o& decision (%ri%bles %nd is toler%nt to so!e

s%!$lin (%ri%bilit"# E- 6eneric %lorith!s46A5 %nd t%bu se%rch 4TS5#

Alorith!-

BASIC GA

%te' 1: Set the iter%tion counter @=>* %nd select %n initi%l $o$ul%tion o&

$ solutions .4>5=)1:;/(8++(9$4>5U

%te' : Run si!ul%tion e)$eri!ents to obt%in $er&or!%nce esti!%tes?4)5 &or %ll $

solutions )4@5 in .4@5#

%te' 4: Select % $o$ul%tion o& $ solutions &ro! those in .4@5 in such %

w%" th%t those with s!%ller ?4)5 (%lues %re !ore li,el"* but not

cert%in* to be selected# Denote this $o$ul%tion o& solutions %s .4@915#

%te' 7: Reco!bine the solutions in .4@915 (i% crosso(er 4which @oins

$%rts o& twosolutions )t4@915 %nd )l4@915 to &or! new solution5 %nd !ut%tion

4which r%ndo!l" ch%nes % $%rt o& solution )i4@915#

%te' 5: Set @=@91 %nd oto ste$ #

The 6A c%n be ter!in%ted %&ter % s$eci'ed nu!ber o& iter%tions* when

little or no i!$ro(e!ent is noted in the $o$ul%tion* or when the

$o$ul%tion cont%ins $ co$ies o& the s%!e solution# At ter!in%tion* the

solution ) th%t h%s the s!%llest ?4)5 (%lue in the l%st $o$ul%tion is

chosen %s the best#

BASIC TS

Ste' 1: set the iter%tion counter @=> %nd the list o& t%bu !o(es to e!$t"#

Select %n initi%l solution ) in #

Ste' : 'nd the solution ) th%t !ini!i+es ?4)5 o(er %ll o& the

neihbours o& ) th%t %re not recei(ed b" t%bu !o(es* runnin

wh%te(er si!ul%tion %re needed to do the o$ti!i+%tion#

8/10/2019 SSM-UNIT 7 8

15/17

Ste' 4: i& ?4)"5Q?4)5 * then )

8/10/2019 SSM-UNIT 7 8

16/17

Ste' 7: u$d%te the list o& t%bu !o(es %nd o to ste$ #

The TS c%n be ter!in%ted when % s$eci'ed nu!ber o& iter%tions

h%(e been co!$leted* when so!e nu!ber o& iter%tions h%s $%ssed

without ch%nin )* or when there %re no &e%sible !o(es# At

ter!in%tion the solution ) is chosen %s best#

Control s%!$lin (%ri%bilit"-o In !%n" c%ses* it will be u$to the user to deter!ine how

!uch s%!$lin will be undert%,en %t e%ch $otenti%l solution#This is % diHcult $roble! in ener%l#

o I& the %n%l"st !ust s$eci&" % ')ed nu!ber o& re$lic%tions $er

solution th%t will be used throuh the se%rch* then %$reli!in%r" e)$eri!ent !ust be conducted#

o Si!ul%te se(er%l desins* so!e %t the e)tre!es o& the

solution s$%ce %nd so!e ne%rer the center# Co!$%re the%$$%rent best %nd %$$%rent worst o& these desins#

o Find the !ini!u! &or the nu!ber o& re$lic%tions re/uired

to decl%re these desins to be st%tistic%ll" sini'c%ntl"diKerent#

o A&ter o$ti!i+%tion run h%s been co!$leted* $er&or! % second

set o& e)$eri!ents on the to$ to 1> desins identi'ed b"the heuristic# Use the co!$%rison techni/ues to riorousl"e(%lu%te which %re the best or ne%r0best o& these desins#

Rest%rtin-o I& $eo$le &%!ili%r with the s"ste! sus$ect th%t cert%in

desins will be ood* be sure to !odule the! %s $ossiblest%rtin solutions &or the heuristic#

RANDOM SEARCH

Let the , $ossible solutions to the o$ti!i+%tion (i% si!ul%tion $roble! bedenoted

)i1*)i(8+(9i! $ro(ides s$eci'c settins &or the ! decision

(%ri%bles# The si!ul%tion out$ut %t )i is denoted ?4)i5 P this could be

the out$ut o& % sinle re$lic%tion or the %(er%e o& se(er%l

re$lic%tions# 8e need to 'nd ) th%t !ini!i+es E4?4)55#

Alorith!-Ste' 1: nitiai6e counter variabes C:i/=; for i=>(3(8+*+ Seect an initia

soution i>* %nd set C4i>5=1#

Ste' : C'oose anot'er soution i from t'e set of a soutions e9ce"t i> insuch % w%" th%t e%ch solution h%s %n e/u%l ch%nce o& beinselected#

Ste' 4: Run si!ul%tion e)$eri!ents %t the two solutions i> an! i to

obtain out"uts Y:i>5

8/10/2019 SSM-UNIT 7 8

17/17

%nd ?:i/+ if Y:i/?Y:i>5* then set i>=i+

Ste' 7: Set C4i>5=C4i>591# I& not done* then oto ste$ # I& done*

then select %s the esti!%ted o$ti!%l solution )i such th%t C4i5 isthe l%rest count#

I& there is % !%)i!i+%tion $roble!* re$l%ce ste$ 7 with

Ste' 4: Run si!ul%tion e)$eri!ents %t the two solutions i> an! i toobtain out"uts Y:i>5 an! Y:i/+ if Y:i/@Y:i>5* then set i>=i+