12. sensitivity analysis in discrete-event simulation using fractional factorial designs

8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs

1/15

Sensitivity analysis in discrete-event simulationusing fractional factorial designsJAB Montevechi*, RG de Almeida Filho, AP Paiva, RFS Costa and AL Medeiros

Instituto de Engenharia de Producao e Gestao, Universidade Federal de Itajuba, Itajuba, Brasil

This paper presents a sensitivity analysis of discrete-event simulation models based on a twofold approach formed by

Design of Experiments (DOE) factorial designs and simulation routines. This sensitivity analysis aim is to reduce the

number of factors used as optimization input via simulation. The advantage of reducing the input factors is that

optimum search can become very time-consuming as the number of factors increases. Two cases were used to illustrate

the proposal: the first one, formed only by discrete variable, and the second presenting both discrete and continuous

variables. The paper also shows the use of the Johnsons transformation to experiments with non-normal response

variables. The specific case of the sensitivity analysis with a Poisson distribution response was studied. Generally,

discrete probability distributions lead to violation of constant variance assumption, which is an important principle in

DOE. Finally, a comparison between optimization conducted without planning and optimization based on sensitivity

analysis results was carried out. The main conclusion of this work is that it is possible to reduce the number of runs

needed to find optimum values, while generating a system knowledge capable to improve resource allocation.

Journal of Simulation advance online publication, 13 November 2009; doi:10.1057/jos.2009.23

Keywords: discrete-event simulation; design of experiments; optimization

1. Introduction

Manufacturing system simulation modelling dates back to at

least the early 1960s (Law and Mccomas, 1998) and is one of

the most popular and powerful tools employed to analyze

complex manufacturing systems (OKaneet al, 2000, Banks

et al, 2005). According to OKane et al(2000), one way toforecast the behaviour of these systems is the use of discrete-

event simulation, which consists of modelling a system where

changes occur at discrete-time intervals. This is appropriate

for manufacturing systems as their behaviour changes in

such a way.

Some manufacturing issues addressed by simulation

include specifying the need and quantity of equipment and

personnel, performance evaluation and evaluation of opera-

tional procedures (Law and Mccomas, 1998). The objectives

of simulation are classified as performance analysis,

capacity/constraint analysis, configuration comparison, op-

timization, sensitivity analysis and visualization (Harrellet al, 2000).

The optimization via simulation deserves special attention.

Harrell et al (2000) define optimization as the process of

trying different combinations of values for the variables that

can be controlled in order to seek for the combination of

values that provides the most desirable output from the

simulation model. However, as the number of variables

increases, the optimization phase becomes more time-

consuming.

To avoid this drawback, sensitivity analysis can be used to

select the variables that are responsible for most of the

variation in the experimental response, eliminating from the

model those variables that are not statistically significant. In

the simulation, sensitivity analysis can be interpreted as a

systematic investigation of simulation outputs according to

the input parameters chosen from the model (Kleijnen,

1998).

In this paper, factorial designs are used to perform a

sensitivity analysis of a simulation model. First, a fractional

factorial design is used to determine the no statistically

significant parameters. Once they are determined, a 2k full

factorial design can be built with the remainders. The

objective is to show how factorial designs can speed up the

simulation optimization to reach the best solution. Two

comparative studies between an all model factors optimiza-tion and another optimization using only the statistically

significant model factors were carried out.

The remainder of this paper is organized as follows: the

next section introduces discrete-event simulation and pre-

sents considerations on performing a simulation. After that,

fractional factorial designs are introduced. And finally, the

considerations on the construction of two simulation

models, the experiments and results analysis are presented.

The paper concludes with some considerations on the

integration between factorial designs and optimization.

*Correspondence: JAB Montevechi, Universidade Federal de ItajubaIEPG, Av. BPS, 1303Caixa Postal: 50, Itajuba, MG, 37500-903,Brasil.E-mail: [email protected]

Journal of Simulation (2009), 115 r 2009 Operational Research Society Ltd. All rights reserved. 1747-7778/09

www.palgrave-journals.com/jos/


2/15

2. Discrete-event simulation

Simulation is the process of designing a model of a real

system and conducting experiments with such a model. This

is done with the purpose of understanding the behaviour

of the system and/or evaluating various strategies for the

operation of the system (Shannon, 1998). Some advantages

of simulation are:

One can simulate systems that already exist as well as

those that are capable of being brought into existence.

Simulation allows one to identify bottlenecks in informa-

tion, material and product flows and to test options for

increasing flow rates.

It allows one to gain insights into how a modelled system

actually works and to understand which variables are

most important to performance.

A significant advantage of simulation is its ability to let

one experiment with new and unfamiliar situations and to

answer what if questions.

Some simulation disadvantages are: model building

requires special training, simulation results can be difficult

to interpret, simulation modelling and analysis can be time

consuming and expensive, the misuse of simulation to solve

problems instead of analytical solution when it is possible or

even preferable, and each run of a stochastic simulation

model produces only estimates of the true characteristics of

model for a particular set of input parameters (Law and

Kelton, 2000; Banks et al, 2005).

In this study, a discrete-event simulation is used. It

consists of modelling a system as it evolves over time by a

representation in which the state variables change instanta-neously at separate points in time (Law and Kelton, 2000).

This methodology is ideal to be applied to manufacturing

systems because they exhibit discrete production changes

(OKane et al, 2000).

According to Strack (1984), Law and Kelton (2000) and

OKaneet al(2000), some characteristics found on problems

to be analyzed that justified the use of simulation are:

real-world systems with stochastic elements cannot be

accurately described by a mathematical model that can

be evaluated analytically;

it is easier to obtain results from a simulation model thanusing an analytical method;

experimentation is impossible or very difficult in a real

world system; and

the need of long-period time studies or alternatives that

physical models do not provide.

3. Optimization via simulation

Optimization via simulation is the process of trying different

combinations of values for variables that can be controlled

in order to seek for the combination of values that provides

the most desirable output from the simulation model

(Harrell et al, 2000). According to Fu (2002), the integration

between optimization and simulation is recent and has been

occurring since the end of the last millennium; the relation-

ship commonly encountered in commercial software is a

subservient one where the optimization routine is an add-on

to the simulation engine. This optimization routine needs the

simulation engine outputs to find the parameter set which

leads to the best solution.

There are several techniques for optimization. Some of

them are based on heuristics. According to Silva and

Montevechi (2004), heuristic techniques accomplish good

solutions and eventually find the optimum solution. How-

ever, it is not possible to state that the solutions found

by these techniques are the best ones, because heuristics do

not test all possible responses.

4. Fractional factorial design

According to Montgomery (2001), when an experiment

involves the study of two or more factors, the most efficient

strategy is the factorial design. In this strategy, the factors

are altered together not one at a time, it means that for each

complete run or replica all possible combination levels are

investigated (Montgomery and Runger, 2003).

When the interest is to study the effects of two or more

factors, factorial designs are more efficient than the one-

factor-at-a-time approach (Montgomery and Runger, 2003).

By a factorial design, it means that in each complete

trial or replication of the experiment all possible combina-

tions of the levels of the factors are investigated. For

example, if there arealevels of factor A andblevels of factor

B, each replicate contains all ab treatment combinations

(Montgomery, 2001). Factorial designs are the only way

to find factor interactions (Montgomery, 2001; Montgomery

and Runger, 2003) avoiding incorrect conclusions when

factors interactions are presented. The factorial design major

issue is the exponential growth of the level combination as

the number of factors increases (Kleijnen, 1998).

The fractional factorial design deals with this by selecting

a subset of all possible points of a factorial design and the

simulation is run only for these points (Law and Kelton,

2000). Therefore, the fractional factorial design providesan alternative to obtain good estimates of main effects and

low order interactions but requires a fraction of computa-

tional work of a full factorial design (Law and Kelton, 2000).

5. System modelling

To accomplish the objective of this paper, two manufactur-

ing cells from different companies have been modelled. The

first one is an application in a Brazilian weapon factory

2 Journal of Simulation Vol.] ], No. ] ]


3/15

whereas the second is an application in a multinational

automotive plant.

5.1. Brazilian weapon factory

The objective of the factory is to increase the throughput by

adding new equipment to the cell. This model is determinis-tic. It means that no source of randomness is considered and

all input data are kept constant (Law and Kelton, 2000;

Banks et al, 2005).

The following information was gathered about the cell:

nine locations;

number of workers available (14 resources); and

the process and how long they last (37 activities), shifts

and part routes.

Promodel 7.0 (http://www.promodel.com/products/

promodel/), which is a discrete-event simulator for manu-

facturing and material handling, was used to build the

models. Figure 1 shows a screen of the models animation in

Promodel 7.0s.

The model verification and validation were performed in

two different ways. Initially, a plant expert analyzed whether

the behaviour of the model would seem reasonable

(Kleijnen, 1995; Sargent, 1998). Then, real historical data

were compared with simulated results.

In order to accomplish a comparison study, an investment

scenario was created. In this scenario, a loan was taken

in order to buy new equipment. The loan payment is made

on a monthly basis and this payment is subtracted from

monthly profit, which is determined by the monthly

production multiplied by unitary profit. The problem is to

select the optimum set of equipment for which the increase

of profit compensates the additional purchase cost of the

equipment. There are nine simulation model parameters

which can be changed. They represent how much equipment

is available to perform a specific task or operation. They

can assume two possible values: 1 or 2. Table 1 presents

these parameters. In this case, the Design of Experiments

(DOE) factors are represented by discrete variables.

The optimum set of equipment is determined by three

approaches. The first one performs several experiments

to identify the main factors and to get a mathematical

model that will be the objective function in an optimization

with Solvers (http://www.solver.com/). After identifying

the statistically significant simulation factors by using a two-

sample t hypothesis test (an usual procedure from any

statistic package), the original fractional factorial design can

be converted into a full factorial, eliminating from the design

Figure 1 Screen of the Brazilian weapon factory models animation.

Table 1 Variable factor assignment for the Brazilian weaponfactory application

Variable Factor Low level ()

High level( )

A Quantity of machines type 1 1 2B Quantity of machines type 2 1 2C Quantity of machines type 3 1 2D Quantity of machines type 4 1 2E Quantity of machines type 5 1 2F Quantity of machines type 6 1 2G Quantity of machines type 7 1 2H Quantity of machines type 8 1 2J Quantity of machines type 9 1 2

Montevechi et alSensitivity analysis in discrete-event simulation using fractional factorial designs 3


4/15

the no statistically significant terms. As these parameters are

also important to the simulation arrangement, despite not

being statistically significant, they can be kept constant in

proper levels. Comparatively, a second approach can be

established by using the main factors identified at the

experiments DOE as input for the optimization via

Simrunners. The optimization software used was Simrunner

3.0s, which uses Genetic Algorithms and is packaged along

with Promodel 7.0s.

Finally, the third approach is performed using all nine

factors in optimization via Simrunners.

5.2. Multinational automotive plant

The objective of this plant is to reduce the production lead

time by adding new operators and reducing setup times.

Production lead time means the time that one batch takes

from the moment raw materials are received to when the

finished products exit. This is a random model, that is, it is

governed by random variables.

The following information was gathered about the cell:

22 locations;

number of workers (six resources); and

the process and their durations (32 activities), shifts and

part routes.

Figure 2 shows a screen of the models animation in

Promodel 7.0.

Similarly to the first application, this model was verified

and validated through process experts analysis and compar-

ison between real historical data and simulated results.

The problem is to minimize the production lead time.

There are 10 simulation model parameters which can be

changed. They represent the number of operators that are

available to perform the cell activities and the setup mean

time of four machines (total setup and partial setup). Table 2

presents these parameters. It can be observed that in this

case, the factors related to the simulation are represented by

both continuous and discrete variables. For sake of

simplicity, the standard deviation of each machine setup

time was kept constant.

The optimum set of parameters is determined by three

approaches similar to the first application.

6. Experimentation

Initially, the experiments are planned to identify the most

statistically significant model factors. After that, the experi-

ments generate an objective function that is used in the

optimization via Solver. Then, these most statistically

significant factors are used as input data for optimization

via Simrunner. Also, this model is optimized using all factors

as input data in order to compare the results.

Figure 2 Screen of the multinational automotive plant models animation.



5/15

6.1. Identifying important factors

According to Law and Kelton (2000), in simulation,

experimental designs provide a way to decide which specific

configurations to simulate before the runs are performed so

the desired information can be obtained with the lowest runs

of simulation. For instance, considering the second applica-

tion where there are 10 factors, if a full factorial experiment

were chosen, it would be necessary 210 1024 runs. There-

fore, a screening experiment must be considered.

Screening or characterization experiments are experiments

in which many factors are considered and the objective is to

identify those factors (if any) that have large effects

(Montgomery, 2001). Typically, screening experiment

involves using fractional factorial designs and it is performed

in the early stages of the project when many factors are likely

considered to have little or no effect on the response

(Montgomery, 2001). According to this author, in this

situation it is usually best to keep the number of factorslevels low.

6.2. Brazilian weapon plant

The experimental design adopted here was a two-level nine-

factor fractional factorial with resolution IV. Resolution IV

means no main effect is aliased with any other main effect or

with any two-factor interaction, but two-factor interactions

are aliased with each other (Montgomery, 2001). The

available resolution IV designs are 2IV93 64 runs and

2IV94 32 runs.

The 2IV94 design was chosen because it has fewer runsthan 2IV

93 design. If necessary, the former design collapses

into a fiveor lessfactor full factorial. As preliminary

studies have shown that five factors (C, D, E, G and H) are

critical to cell performance, this design works well. The

generators for this design are F7BCDE, G7ACDE,

H7ABDE and J7ABCE, and the defining relations

are IBCDEFACDEGABDEHABCEJABFG

ACFH ADFJBCGH BDGJCDHJ DEFGH

CEFGJBEFHJAEGHJABCDFGHJ. Table 1 shows

the factor assignment to the variables of the design.

Table 3 shows the design matrix for principal fraction with

the results obtained for each run. Analyzing the response

variable (Profit) contained in this table, it can be observed

that this response variable is not normal. The aforemen-

tioned response (Profit) was determined multiplying the

monthly production by a unitary profit. Mathematically, theresulting output variable can be interpreted as a number of

units produced in a month, which is typically a Poisson

distribution case. According to many researchers (Bisgaard

and Fuller, 1994; Lewis et al, 2000; Montgomery, 2001)

when using counts as the experimental response, the

assumption of constant variance made with all standard

analysis is violated. Then, a common method for dealing

with this problem is to transform the data before the

significance analysis, so that the assumption of constant

variance is more probable. To verify if the experimental data

really follows a Poisson distribution as discussed early, a

Goodness-of-fit test based on the chi-square distribution can

be applied (Haldar and Mahadevan, 2000). In this test, the

null hypothesis (H0) indicates the observed data follows a

Poisson distribution. To obtain the chi-square statistic it is

necessary to calculate the probability density function for

Poisson distribution, which can be written as Equation (1):

PX ni ellni

ni! X 0; 1; 2;4 1

To accomplish with the aims of the chi-square test, the

expected frequency Eiof the data, can be calculated as:

Ei N ellni

ni!

2

In Equation (2), Nis the total number of observations inthe sample (N 32 in this case), l is the mean of Poisson

distribution andniis the number of observation counted for

each class.

For the specific case of the Goodness-of-fit test, where there

is only one column to the data, the chi-square statistic becomes

w2TXri1

ni Ei2

Ei4w2m1k;1a 3

In the Equation (3), ni is the frequency observed in the

sample, m is the used number of classes used in distribution

Table 2 Variable factor assignment for the multinational automotive plant application

Variable Factor Low level () High level( )

A Quantity of operators 2 4B Mean time of the processing of the operators 2 1C Mean time of the total setup to machine_01 109 83D Mean time of the partial setup to machine_01 39 21

E Mean time of the total setup to machine_02 154 105F Mean time of the partial setup to machine_02 80 55G Mean time of the total setup to machine_03 152 115H Mean time of the partial setup to machine_03 55 35J Mean time of the total setup to machine_04 74 46K Mean time of the partial setup to machine_04 49 28



6/15


7/15

significance of the main and the interaction effects using the

conventional bilateral t-test or ANOVA. The standard

analysis procedure for an unreplicated two-level design is

the normal plot of the estimated factor effects. However,

these designs are so widely used in practice that many formal

analysis procedures have been proposed to overcome the

subjectivity of the normal probability (Montgomery, 2001).

Ye and Hamada (2001), for instance, recommend the use of

Lenths method, a graphical approach based on a Pareto

Chart for the error term. If the error term has one or more

degrees of freedom, the line on the graph is drawn at t, where

t is the (1a/2) quantile of t-distribution with degrees of

freedom equal to the (number of effects/3) (as shown in

Figure 4). The vertical line in the Pareto Chart is the margin

of error, defined as ME tPSE. Lenths pseudo standard

error (PSE) is based on sparsity of the effects principle,

which assumes the variation in the smallest effects is due to

the random error. To calculate PSE the following steps are

necessary: (a) calculates the absolute value of the effects; (b)

calculates S, which is 1.5median of the step (c); calculatesthe median of the effects that are less than 2.5 S and (d)

calculates PSE, which is 1.5 median calculated in step (c).

Examining the Pareto Chart in Figure 4, it can be noted

that at a significance level of 10% only factor C (Quantity of

equipment 3) and the interactions BC and CE are significant.

According to Montgomery (2001), if the experimenter can

reasonably assume that certain high-order interactions are

negligible, information on the main effects and low-order

interactions may be obtained. Otherwise, when there are

several variables, the system or process is likely to be driven

primarily by some of the main effects and low-order

interactions. Therefore, taking into consideration that

factors A, D, F, G, H and J are not significant, though

they are necessary to the simulation process, in the next step

of the experimentation process these factors must be kept at

low level (1), once these levels present higher profit values.

The factors B and E were kept in the set of meaningful input

variables because they are placed in the interactions BC and

CE, respectively. The main effects for transformed response

are shown in Figure 5. It is noticed that factors B (Quantity

of equipment 2) and C (Quantity of equipment 3) have

positive effect. These factors increase profit when they are

increased (high level). The other factors have negative effects

that decrease profit when they are increased (high level). The

analysis of the interactions in a fractional factorial is not

recommended as the confounding among the effects are

tremendous. However, in a strict sense, it is possible to use

the aliases (confounding) among the main and interaction

effects to explain why only the factors B, C and E were

chosen to compose the new design. Examining the con-founding among the statistically significant terms C, BC and

CE and the other factors and interactions of the 2IV94

fractional factorial design, it is possible to notice that the

factor C is aliased with a three-factor interactions AFH,

BGH and DHJ, which may be neglected according to the

resolution of the chosen design and sparsity of the effects

principle (Montgomery, 2001). The two-way interaction CE

is aliased with three-way interactions ABJ, ADG, BDF and

FGJ, which may also be discarded. The interaction BC is

aliased with the two-way interaction GH and the three-way

Percent

400000300000200000

99

90

50

10

1

N 32

AD 2,809

P-Value


8/15

interactions AEJ and DEF. Neglecting these three-way

interactions and considering that GH is formed by two weak

effects (G and H), besides being aliased with three-way

interactions, there is no reason to believe that any of these

factors is statistically significant. The alias structure used in

this analysis is available in many statistics packages.

As only factors B, C and E have presented some evidence

of significance, the 2IV94 design can be converted into a

replicated 23 full factorial design. Figures 6 and 7 show the

analysis for this new design, where it is possible to notice

that factors B and C must be changed to the high level (2) to

maximize the transformed response and, consequently, the

Term

Effect

ABBDAGAFEHAD

EEGAEEJEFDE

AEFCJACAHCDBH

HB

AJGDFJA

BECEBC

C

1.41.21.00.80.60.40.20.0

0.497

Lenth's PSE = 0.275013

Figure 4 Pareto Chart for 2IV94 Fractional design with significance level a 10%.

MeanofTransformedresponse

0.5

0.0

-0.5

1-1 1-1 1-1

0.5

0.0

-0.5

1-1 1-1 1-1

0.5

0.0

-0.5

1-1 1-1 1-1

A B C

D E F

G H J

Figure 5 Main effects plot for transformed Profit.



9/15

process Profit. Although the main effect E is not statistically

significant, observing the interaction plot of Figure 7, it is

clear that choosing factor E placed in the high level

combined with factors B and C in their respective higher

levels, the response increase considerably. However, in order

to check if it is the best solution, an optimization using these

three factors is performed. Also, an optimization using all

nine factors is performed to compare their performances.

6.3. Multinational automotive plant

The experimental design adopted here was a two-level

10-factor fractional factorial with resolution IV. The avail-

able resolution IV designs are 2IV93 64 runs and 2IV

94 32

runs. The 2IV93 design was chosen because it has a higher

number of experiments (Montgomery, 2001). Table 2 shows

the factor assignment to the variables of the design.

Term

Standardized Effect

E

B

BCE

BE

CE

BC

C

6543210

1,711

Figure 6 Pareto Chart for full factorial design with significance levela 10%.

MeanofJohnson

0.5

0.0

-0.51-1 1-1

1-1

0.5

0.0

-0.5

B C

E

B

C

E

1-1 1-1

1

0

-1

1

0

-1

B-11

C-11

Main Effects Plot (data means) for Johnson Interaction Plot (data means) for Johnson

Figure 7 Factorial and interaction plots for 2

3

full factorial design.



10/15

The design matrix for main fraction with the results

obtained was omitted because it is not the centre of interest

of this paper. Analyzing the response variable (production

lead time), it can be verified through a normality test that

this response variable is normal.

Similar to the first application, the 2IV93 fractional factorial

design used here is unreplicated. Examining the Pareto Chart

in Figure 8, it can be noted that at a significance level of 5%

only factors A (number of operators), E (mean time of the

total setup for machine 2) and the interactions AC, FH and

EG are significant. Therefore, taking into consideration that

factors B, C, D, G, J and K are not significant, though they

are necessary to the simulation process, in the next step of

the experimentation process these factors must be kept at

low level. Factors F and H were kept in the set of statistically

significant input variables because they are placed in the

interactions FH.

The aliases (confounding) among the main and interaction

effects is used to explain why only factors A, E, F and H

were chosen to compose the new design. Examining the

confounding among the statistically significant terms A, E,AC, FH and EG and the other factors and interactions of

the 2IV93 fractional factorial design, it is possible to notice that

factor C is aliased with a three-factor interaction BGH and

with a four-factor interactions ABEK, ADFH, BDFG and

EGHK, which may be neglected according to the resolution

of the chosen design and sparsity of the effects principle

(Montgomery, 2001). Similarly, factor E is aliased with a

four-factor interaction. The two-way interaction AC is

aliased with three-way interactions BEK and DFH and

with four-factor interactions BCGH and EFGJ, which may

also be discarded. The interaction EG is aliased with the

three-way interactions CHK and DHJ and with the four-

factor interactions ACFJ and ADFK, neglecting these three-

way and four-way interactions. Similarly, the two-way

interaction FH is aliased with a three-way and a four-factor

interaction.

The main effects for production lead time are shown in

Figure 9. It is noticed that factor A (number of operators)

has a strong negative effect on production lead time. This

factor decreases the production lead time when it is increased

(high level). Factor E (mean time of the total machine 2

setup) also reduces the production lead time when it is

increased (high level).

As only factors A, E, F and H have presented some

evidence of significance, the 2IV93 design can be converted in a

replicated 24 full factorial design. Figures 10 and 11 show the

analysis for this new design, where it is possible to notice

that factors A, E, F and H must be changed to the high level

to minimize the lead time.

This finding can be explained in practice as follows: the

company needs to reduce production lead time to increaseoutput and to answer to its customers, then it is possible to

change 10 factors to reach this objective, however, only four

factors are significant to reduce production lead time. So, the

company can apply its efforts only by adding new operators

and by reducing three setup times of two different machines

(2 and 3) that it will reach its objective.

A mathematical model can be obtained from the full

factorial analysis, as is shown by Equation (1), where the

characters represent the coded values of the respective

factors. A minimum production lead time can be calculated

Term

Effect

ADGFJ

CEAEH

BCAFDGHJBK

DAEF

EKEH

AGJAH

ACJCFBF

AGKBJ

BFGEJH

ADEGFH

EAC

A

1.81.61.41.21.00.80.60.40.20.0

0.279

Factor

EF

GH

J

Name

TS_04K

ABCD

Lenth's PSE = 0.134063

num_operpro_timeTS_01PS_01

TS_02PS_02

TS_03PS_03

PS_04

Figure 8 Pareto Chart for 2IV104 fractional design with significance level a 5%.

10 Journal of Simulation Vol. ] ], No. ] ]


11/15

by using linear programming via Solver from Excel,

considering Equation (4) as the objective function and the

values shown in Table 2 for the restrictions of the statistically

significant factors (A, E, F and H). This procedure points

out the minimum production lead time as 8.59 h. The values

to variables A, E, F and H are shown in Table 5.

Leadtime 9:8656 0:8325A

0:1722E 0:0087F

0:1278G 0:1478FG

4

In order to make a comparison between the current

approach and the genetic algorithms (GAs) approach, an

MeanofProductionLead

Time

10.4

9.6

8.8

42 12 83109 2139

10.4

9.6

8.8

105154 5580 115152 3555

10.4

9.6

8.84674 2849

num_oper pro_time TS_01 PS_01

TS_02 PS_02 TS_03 PS_03

TS_04 PS_04

Figure 9 Main effects plot for lead time.

Term

Standardized Effect

BCD

BC

ACD

AB

C

ABCD

ABD

AC

ABC

BD

AD

D

CD

B

A

121086420

2.01

Factor

PS_03

Name

ABCD

num_operTS_02PS_02

Figure 10 Pareto Chart for full factorial design with significance levela 5%.



12/15


13/15

6.5. Optimization using all factors

Brazilian weapon factory. In this optimization phase, the

nine parameters presented in Table 1 are selected as inputs.

The values these parameters can assume are 1 or 2. The

objective function is to maximize profit, as presented

earlier.

After 98 runs, the Simrunner stops the search. The bestresult found is 411.327, which corresponds to experiment 55

as presented at Figure 13 and the correspondent parameter

values are presented in Table 7.

Multinational automotive plant. The 10 parameters

presented in Table 2 are selected as inputs. The values

these parameters assumed were also shown in Table 2. The

objective function is to minimize the production lead time,

as presented earlier.

After 2571 runs (3 h 20 min), the Simrunner stops the

search. The best result found is 7.42 h and the correspondent

parameter values are presented in Table 8.

6.6. Result analysis

Brazilian weapon factory. Table 9 shows the results

obtained by the three procedures. The three procedures

lead to the same results indicating that there is coherency

among them. Taking into consideration the number of runs

necessary to optimize the model, it is clear the advantage of

determining previously the main parameters and then

proceeding the optimization using them instead of proceed-

ing with the optimization using all factors. The former

demanded 32 8 40 runs to obtain the best result against

the 98 runs from the latter, a reduction of 59% in the

number of runs. Considering only the runs used in the

Factorial Design, 32 runs versus 98 runs from optimization

using all factors, the reduction is about 67%.

For this application, since the optimization factor

levels and the Factorial Design levels are equal, the

optimization using three factors seems meaningless.

However, other applications where the optimization factors

could have several levels or even were represented by

continuous variables, the Factorial Designs would only

identify the main factors without specifying their optimum

values.

Table 7 Best solution for optimization using all factors for the

Brazilian weapon factory application

Parameter Value

Neq_Op050 1Neq_Op052 1Neq_Op070 1Neq_Op080 2Neq_Op082 2Neq_Op100 1Neq_Op110 1Neq_Op120 2Neq_Op170 1

Table 8 Best solution for optimization using all factors for themultinational automotive plant application

Parameter Value

Quantity of operators 4Processing time of the operators N(1; 0.30)Total setup time of the machine_01 N(102; 42)Partial setup time of the machine_01 N(21; 12)Total setup time of the machine_02 N(126; 22)Partial setup time of the machine_02 N(55; 46)Total setup time of the machine_03 N(119; 41)

Partial setup time of the machine_03 N(35; 4)Total setup time of the machine_04 N(54; 33)Partial setup time of the machine_04 N(33; 15)

Table 9 Results from the three procedures for the Brazilianweapon factory application

Parameter Design of experiments

Optimizationusing three

factors

Optimizationusing allfactors

A 1 1 (*) 1

B 1 1 (*) 1C 1 1 (*) 1D 2 2 2E 2 2 2F 1 1 (*) 1G 1 1 (*) 1H 2 2 2J 1 1 (*) 1Result (Profit) 411 327 411 327 411 327

Number of runs 32 8 98

Note: The parameters identified with (*) were not used as input foroptimization. They were kept at low level (1).

Figure 13 Performance measures plot for optimization usingall factors.



14/15

Multinational automotive plant. Table 10 summarizes

the results obtained by the three procedures. The optimiza-

tion with all factors generates the lowest production lead

time and spent 3.2 h of optimization, whereas the optimiza-

tion with only the statistically significant factors lead to a

value to the production lead time 12% higher (52 min), by

spending 0.3 h in optimization plus 3 h in experimentation

through factorial designs. At last, the optimization through

the mathematical model obtained from the factorial design

and solved by Solver lead to a value to the production

lead time 16% higher (70 min) than the first procedure

(using all factors). It took 0.2 h in optimization plus 3 h in

experimentation through factorial design. Taking into

account the objective of the analysis, that is, to find the

best production lead time, the best procedure is

the optimization with all the factors. By doing it this way,

the computational time spent is practically the same in

comparison to the other procedures.

Additionally, the human effort spent in the optimization

with all factors is the lowest when compared to building 64

experiments (simulation models) in the factorial designs.

However, in practice, to get the results indicated by theoptimization with all factors, it would be necessary to

modify eight setup times, one processing time and the

number of operators. It means that the company would

spend more money and time by changing all these factors

than by changing only the four statistically significant

factors, that is the result of the factorial designs (three setup

times and the number of operators). Thus, the factorial

designs can be used to point out the main parameters

and then the optimization can be used to specify their

optimum values.

7. Conclusions

The objective of this work was to show how a sensitivity

analysis using Factorial Designs can help the simulation

optimization to reach the best solution. Initially, to

accomplish this objective a fractional factorial design was

used to identify the most statistically significant effects of

two models. As the experimental response variable of thefirst application does not follow a normal distribution, it was

necessary to apply a transformation to make it normal. As

the response variable follows a Poison distribution, the

Johnson transformation was used. The analysis was

performed using the Lenths method because the fractional

factorial design was unreplicated which made the bilateral

t-test or ANOVA not possible to assess the significance

of the main and interactions effects.

Afterwards, three optimization trials/tests were carried

out: two using the factors previously identified, and the other

using all factors of the model. For these applications, it is

clear the advantage of determining previously the mainfactors and then proceeding the optimization using them

(Solver or Simrunner) instead of proceeding with the

optimization using all factors. Beyond this discussion, it

must be pointed out that the DOE approach improves the

manufacturing system understanding, generating further

knowledge about the importance and significance of each

resource used, which in turn favors the improvement of the

decision-making process. More than how many resources

are needed to optimize a system, improving its productivity,

increasing its profits and reducing costs, despite of the

relative time spent in the construction of the models,

this twofold approach (DOE/Simulation) elucidates how

the resource can be efficiently changed and employed.

AcknowledgementsThe authors acknowledge CNPq, PadTec OpticalComponents and Systems, CAPES and FAPEMIG for the support tothis research study.

References

Banks J, Carson JS, Nelson BL and Nicol DM (2005). Discrete-

Event System Simulation4th edn, Prentice-Hall: New Jersey.

Bisgaard S and Fuller H (1994). Analysis of factorial experimentswith defects or defectives as the response. Quality Eng 7(2):

429443.

Chou Y, Polansky AM and Mason RL (1998). Transforming

nonnormal data to normality in statistical process control.

J Qual Technol30: 133141.

Fu MC (2002). Optimization for simulation: Theory vs. practice.

INFORMS J Comput 14(3): 192215.

Haldar A and Mahadevan S (2000). Probability, Reliability and

Statistical Methods in Engineering Design1st edn, John Wiley &

Sons: New York.

Harrell C, Ghosh BK and Bowden R (2000). Simulation Using

ProModel3rd edn, McGraw-Hill: Boston.

Table 10 Results from the three procedures for the multi-national automotive plant application

Parameter Design of experimentsand solver

Optimizationusing four

factors

Optimizationusing allfactors

A 2 3 4

B 2 (*) 2 (*) 1C 109 (*) 109 (*) 102D 39 (*) 39 (*) 21E 105 153 126F 80 73 55G 152 (*) 152 (*) 119H 35 48 35J 74 (*) 74 (*) 54K 49 (*) 49 (*) 33Lead Time (hour) 8.59 8.28 7.42

Number of runs 64 213 2571Computational time(hour)

3 0.2 3.2 3 0.3 3.3 3.2

Note: The parameters identified with (*) were not used as input for

optimization. They were kept at low level.

14 Journal of Simulation Vol. ] ], No. ] ]


15/15

Kleijnen JPC (1995). Theory and methodology: Verification

and validation of simulation models. Eur J Opl Res 82:

45162.

Kleijnen JPC (1998). Experimental design for sensitivity analysis,

optimization, and validation of simulation models. In: Banks J

(ed). Handbook of Simulation 1998. John Wiley & Sons: New

York, pp 173223.

Law AM and Kelton WD (2000). Simulation Modeling and Analysis

3rd edn, McGraw-Hill: New York.Law AM and McComas MG (1998). Simulation of manufacturing

systems. In: Medeiros DJ, Watson EF and Manivannan MS

(eds). Proceedings of the 1998 Winter Simulation Conference.

Institute of Electrical and Electronics Engineers; Piscataway,

New Jersey, pp 4952.

Lewis SL, Montgomery DC and Myers RH (19992000). The

analysis of designed experiments with non-normal responses.

Quality Eng 12(2): 225243.

Montgomery DC (2001). Design and Analysis of Experiments 5th

edn, John Wiley & Sons: New York.

Montgomery DC and Runger GC (2003). Applied Statistics and

Probability for Engineers 2nd edn, John Wiley & Sons: New

York.

OKane JF, Spenceley JR and Taylor R (2000). Simulation as an

essential tool for advanced manufacturing technology problems.

J Mater Process Tech 107: 412424.

Sargent R.G (1998). Verification and validation of simulation

models. Proceedings of the 1998 Winter Simulation Conference.

Shannon RE (1998). Introduction to the art and science of

simulation. In: Medeiros DJ, Watson EF and Manivannan

MS (eds).Proceedings of the 1998 Winter Simulation Conference.

Institute of Electrical and Electronics Engineers; Piscataway,New Jersey, pp 714.

Silva WA and Montevechi JAB (2004). Verificacao do custeio de

uma ce lula de manufatura usando simulacao e otimizacao. In:

Anais do XXXVI Simposio Brasileiro de Pesquisa Operacional

SBPO: Sao Joao Del Rei, Minas Gerais. CD-ROM.

Strack J (1984). GPSS: modelagem e simulacao de sistemas. LTC:

Rio de Janeiro.

Ye KQ and Hamada M (2001). A step-down Lenth method for

analyzing unreplicated factorial designs. J Qual Technol33(2):

140153.

Received 27 September 2007;

accepted 29 July 2009 after two revisions


12. sensitivity analysis in discrete-event simulation using fractional factorial designs

Documents