finding feasible systems in the presence of constraints on multiple performance measures

13

Finding Feasible Systems in thePresence of Constraints on MultiplePerformance Measures

DEMET BATURUniversity of Nebraska-LincolnandSEONG-HEE KIMGeorgia Institute of Technology

We consider the problem of finding a set of feasible or near-feasible systems among a finite numberof simulated systems in the presence of constraints on secondary performance measures. We firstpresent a generic procedure that detects the feasibility of one system in the presence of oneconstraint and extend it to the case of two or more systems and constraints. To accelerate theelimination of infeasible systems, a method that reuses collected observations and its variance-updating version are discussed. Experimental results are presented to compare the performanceof the proposed procedures.

Categories and Subject Descriptors: D.2.7 [Software Engineering]: Distribution, Mainte-nance, and Enhancement—Documentation; H.4.0 [Information Systems Applications]: Gen-eral; I.7.2 [Document and Text Processing]: Document Preparation—Languages and systems,Photocomposition/typesetting

General Terms: Algorithms, Design, Experimentation

Additional Key Words and Phrases: Multiple performance measures, ranking and selection,stochastic constraints

ACM Reference Format:Batur, D. and Kim, S.-H. 2010. Finding feasible systems in the presence of constraints on multipleperformance measures. ACM Trans. Model. Comput. Simul. 20, 3, Article 13 (September 2010),26 pages. DOI = 10.1145/1842713.1842716 http://doi.acm.org/10.1145/1842713.1842716

This material was supported by the National Science Foundation under Grant NumbersDMI-0400260 and DMI-0644837. A part of this article is published in Batur and Kim [2005].Authors’ addresses: D. Batur, Industrial and Management Systems Engineering, University ofNebraska-Lincoln, Lincoln, NE 68588-0518; S.-H. Kim, H. Milton Stewart School of Industrialand Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332-0205; email:[email protected] to make digital or hard copies of part or all of this work for personal or classroom useis granted without fee provided that copies are not made or distributed for profit or commercialadvantage and that copies show this notice on the first page or initial screen of a display alongwith the full citation. Copyrights for components of this work owned by others than ACM must behonored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers,to redistribute to lists, or to use any component of this work in other works requires prior specificpermission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 PennPlaza, Suite 701, New York, NY 10121-0701 USA, fax +1 (212) 869-0481, or [email protected]© 2010 ACM 1049-3301/2010/09-ART13 $10.00DOI 10.1145/1842713.1842716 http://doi.acm.org/10.1145/1842713.1842716

ACM Transactions on Modeling and Computer Simulation, Vol. 20, No. 3, Article 13, Pub. date: September 2010.

13:2 • D. Batur and S.-H. Kim

1. INTRODUCTION

Ranking and selection (R&S) mainly focuses on finding the best among a finitenumber of simulated systems—usually between 2 and 500—with some guar-antee about correctness. In this article, the best system means the one withthe largest or smallest expected primary performance measure. The finding-the-best problem has been actively studied in the simulation community (seeGoldsman et al. [2005] and Kim and Nelson [2006b] for recent literature). How-ever, very little work has been done in finding the best system in the presenceof secondary performance measures. Consider the following two problems.

Problem 1. The analyst wants to select a production schedule for a manu-facturing system that yields the largest expected throughput among a numberof different schedules, and the decision maker wants to keep the expected leadtime in the system bounded (smaller than or equal to some constant) at thesame time.

Problem 2. A customer service center (CSC) is a warehouse that processesboth Internet and business-to-business transactions. Orders are received toCSC every morning at 11am and expected to be shipped within 24 hours. Thereare four main tasks to be done in specific time periods: in-bound receivingfrom 11pm to 11am, batch refill from 8am to 10am, order processing (calledproduction) from 10am to 10pm, and loading from 5pm to 2am. These tasks aremainly done by pallet jacks, reach trucks, and stock pickers. As those machinesare expensive, the analyst wants to determine the optimal amount of equipmentamong a number of alternatives that minimizes the cost of equipment whilemaking sure that on average 95% of the orders are processed within 24 hours.

The primary performance measure and constraint on a secondary perfor-mance measure of Problem 1 are stochastic in that the underlying system iscomplex enough to require simulation to evaluate the primary and secondaryperformance measures. On the other hand, the primary performance measureof Problem 2 (the total cost of equipment) is deterministic but its constraintsare stochastic. To solve Problem 1, we need a procedure that can jointly handlestochastic primary performance measure and stochastic constraints on sec-ondary performance measures at the same time. To solve Problem 2, we onlyneed a procedure for a feasibility check.

Multiple performance measures are considered in Butler et al. [2001] andSantner and Tamhane [1984]. However, the methods presented in these twopapers are either hard to implement in practice or specialized to the mean asa primary performance measure and the variance as a secondary performancemeasure. Andradottir et al. [2005] and Andradottir and Kim [2010] considerone primary and one generic secondary performance measure and impose aconstraint on the secondary performance measure. They present an R&S proce-dure that determines the feasibility of systems in the presence of one stochasticconstraint with high probability of being correct (which can solve Problem 2)and combine the procedure with a finding-the-best procedure to identify thebest feasible system (which can solve Problem 1). Problems 1 and 2 have onlyone stochastic constraint but it is possible that there are multiple stochastic


Finding Feasible Systems in the Presence of Constraints • 13:3

constraints. Thus, the theory needs to be further extended to the case of mul-tiple stochastic constraints.

In this article, we provide procedures for determining a set of feasible or near-feasible systems among a finite number of simulated systems in the presence ofmultiple stochastic constraints. We assume that the number of secondary per-formance measures, (thus the number of stochastic constraints), is relativelysmall (no more than five). If the size of the number of secondary performancemeasures times the number of systems is too large (say, larger than 500, whichis a typical upper bound for fully sequential R&S procedures; see Kim and Nel-son [2001]), then methods other than R&S procedures should be considered.The proposed procedures in this article can, (1) solve an optimization problemwhen the objective function is deterministic but the constraints are stochastic,(2) serve as critical steps in finding the best feasible system for a constrainedR&S problem, or (3) check which systems meet certain performance criteria.The problem of finding the best feasible system when both the objective functionand constraints are stochastic, can be addressed by combining any resultingprocedure from this article with a finding-the-best procedure under the frame-work of Andradottir and Kim [2010], which is a topic of ongoing research.

Let F be an R&S procedure that checks the feasibility of one system in thepresence of one stochastic constraint with a prespecified probability of correctdecision (PCD). Any such F can easily be extended to the case of multiplesystems and constraints by the use of the Bonferroni inequality. We call theextended procedure FB. Unfortunately, FB tends to be conservative, and thisconservatism becomes more serious as the number of systems or constraintsincreases. To lessen this problem, we present a screening procedure that accel-erates the elimination of infeasible systems, namely, the accelerated procedure,FA. The idea is to reuse collected observations by taking a linear combinationof the observations across the stochastic constraints from each system.

A variance-updating R&S procedure updates variance estimates as more ob-servations are obtained. In Kim and Nelson [2006a], a variance-updating proce-dure known as KN++ is presented. This procedure is specifically designed for asteady-state simulation with a single replication design, uses raw observations,which are likely correlated, from the single replication as basic observations,and is shown to be asymptotically valid. In Malone et al. [2005], KN++ is ap-plied to independent and identically distributed (IID) normal data. They foundthat, (1) the procedure shows significant savings, from 20% up to 80%, in termsof the number of observations compared to a corresponding nonupdating ver-sion; (2) the procedure is still asymptotically valid; (3) the procedure does notguarantee a correct selection with a prespecified probability for a finite samplesize even for IID normal data; but (4) the degradation from the nominal prob-ability requirement is not significant. To further improve the efficiency of theFA procedure, we develop a variance-updating version of it.

The article is organized as follows. Section 2 formulates our problem andgives notation and definition. Section 3 provides the generic algorithms ofF , FB, and FA with and without variance updating. Example procedures arepresented in Section 4 by extending Algorithm I of Andradottir et al. [2005]and Andradottir and Kim [2010] to the case of multiple constraints. Finally, we



compare the performance of the proposed procedures by experimental resultsbased on multivariate random variables with normal and Bernoulli marginaldistributions and a simple queueing model in Section 5, followed by a conclusionin Section 6.

2. PROBLEM

Our problem is to determine a set of feasible or near-feasible systems amongk simulated systems that satisfy s stochastic constraints. Let Yi�j for i =1, 2, . . . , k, � = 1, 2, . . . , s, and j = 1, 2, . . . denote an observation from thejth replication associated with the �th performance measure (or the �th con-straint) of the ith system. Let Yij = (Yi1 j, Yi2 j, . . . , Yisj)T be the vector of thejth observations across all s performance measures of system i where cT rep-resents the transpose of c. The expected performance measures of system i aredefined as yi = E[Yij] = (yi1, yi2, . . . , yis)T , where E[Yi�j] = yi� for � = 1, 2, . . . , sand j = 1, 2, . . . . We make the following assumption on Yij for i = 1, 2, . . . , k,j = 1, 2, . . .:

ASSUMPTION 1.

Yij =

⎡⎢⎢⎢⎢⎣

Yi1 j

Yi2 j

...Yisj

⎤⎥⎥⎥⎥⎦

IID∼ MN (yi, �i) ,

where ∼ and MN represent ‘are distributed as’ and ‘multivariate normal’, re-spectively, and �i is the variance-covariance matrix of Yij.

Throughout the article, we write vectors in boldface, random variables inupper case, and their realizations in lower case.

If the observations from the simulation output do not satisfy Assumption 1,then Assumption 1 might still approximately hold, such as in certain meanapplications or when the method of batch means is used. Also, observationsassociated with different performance measures from a system are likely tobe correlated in reality, such as total inventory cost versus the total number ofback orders for an inventory system. Therefore, the assumption of multivariatenormal random variables is plausible. The vectors Yij and Yν j can be depen-dent for i �= ν due to the use of common random numbers (CRN). It is knownthat CRN often help increase the efficiency of statistical selection procedures.However, although the proposed procedures for a feasibility check in this ar-ticle can handle CRN, their efficiency will not be benefited at all by the useof CRN since a feasibility check does not require comparisons among systems.Nevertheless, we consider the case of CRN as well as the independent casebecause the proposed procedures may eventually be embedded into a proce-dure for finding the best feasible system, which requires comparison amongsystems. If an experimenter runs a study to check only the feasibility ofsystems, then a feasibility check procedure for the independent case would bedesirable.



Note that probability and variance are the expectations of an indicator ran-dom variable and sample variance, respectively. Thus, any constraint on aprobability or variance can be expressed in terms of an expectation. Morespecifically, 0–1 observations will be basic observations when there is a con-straint on a probability of IID observations. For a constraint on a variance ofIID observations, batching is needed and sample variances from each batchwill be used as basic observations. One might raise the non-normality of 0–1observations or sample variances. This problem could be lessened by batchingof 0–1 observations or sample variances, as shown in Kim and Nelson [2001].That is, the batch means of a number of 0–1 observations or sample variancescould be taken as basic observations. This method may not be desirable espe-cially for variances because it would require batching of batching. However,Kim and Nelson [2001] and Malone et al. [2005] show that fully sequentialR&S procedures tend to be robust to non-normality. As our procedures are fullysequential, it is possible that our procedures are also robust to non-normalityand could be applied directly to 0–1 observations or sample variances. This isdiscussed in more detail in Section 5.2. Handling constraints on a probabilityor variance of correlated observations (e.g., raw observations in a single repli-cation of steady-state simulation) is more complicated and difficult. We do notdiscuss it in this article, as our interest is more on IID observations.

We define feasible systems as those with the vector of mean performancemeasures smaller than or equal to a constant vector q′ = (q′

1, q′2, . . . , q′

s)T .

More specifically, system i is feasible if yi ≤ q′. Unfortunately, for stochasticsystems, it is impossible to guarantee identifying all feasible systems thatsatisfy s stochastic constraints with a finite number of observations. Instead,Andradottir et al. [2005] and Andradottir and Kim [2010] introduce a tolerancelevel which is similar to the indifference-zone parameter—minimum differenceworth detecting (see Kim and Nelson [2006b]). We adopt the same approach.For each constraint �, � = 1, 2, . . . , s, the decision maker gives a range aroundq′

�, say, (q−� , q+

� ) such that q−� ≤ q′

� ≤ q+� and q−

� < q+� . Let q− = (q−

1 , . . . , q−s )T

and q+ = (q+1 , . . . , q+

s )T . Then the following three regions are defined for theconstraints we consider.

—yi ≤ q−. This is the desirable region. If a system is in this region, then it isfeasible.

—(yi < q+)\(yi ≤ q−). This is the acceptable region. If a system is in this region,it is either feasible or infeasible and can be declared feasible or infeasible byour procedure regardless of its true feasibility.

—(yi1 ≥ q+1 ) ∪ (yi2 ≥ q+

2 ) ∪ · · · ∪ (yis ≥ q+s ). This is the unacceptable region. If a

system is in this region, then it is infeasible and should be eliminated.

Furthermore, we define the following three sets for the constraints under con-sideration:

SD = the set of all desirable systems;

SA = the set of all acceptable systems; and

SU = the set of all unacceptable systems.



Fig. 1. Desirable (D), acceptable (A), and unacceptable (U) regions when there are two stochasticconstraints.

For given q−� and q+

� , we define q� and ε� as q� = (q−� + q+

� )/2 and ε� =(q+

� − q−� )/2, respectively. Then q−

� = q� − ε� and q+� = q� + ε�. The parameter q�,

called the target value of the �th constraint, behaves as a cutoff point betweenfeasible and infeasible systems for the �th constraint. The parameter ε� is thetolerance level of the �th constraint, indicating how much we are willing tobe off and above from q�. Then q = (q1, q2, . . . , qs)T and e = (ε1, ε2, . . . , εs)T

represent the vectors of target values and tolerance levels for the s constraints,respectively.

Figure 1 shows the desirable (D), acceptable (A), and unacceptable (U) re-gions in terms of q� and ε� for � = 1, 2 when there are two stochastic constraints.Our procedures will be presented in terms of q� and ε�.

Finally, a correct decision (CD) is defined as the event for which a procedureselects a set F such that SD ⊆ F ⊆ (SD∪SA), and a statistically valid procedureshould guarantee the following probability statement.

PCD ≡ Pr{CD} = Pr{SD ⊆ F ⊆ (SD ∪ SA)} ≥ 1 − α,

where 1 − α is the nominal confidence level.

3. GENERIC ALGORITHMS

In this section, we present generic algorithms for checking feasibility of k sys-tems when there are s stochastic constraints. The purpose of this section isto provide a framework that helps extend any procedure that works for onesystem and one constraint to more general cases.



Fig. 2. A triangular continuation region for the constraint y ≤ q′.

3.1 Multiple Feasibility Check Procedure (FB)

In this section, we first present a generic algorithm of F for the case of onesystem and one constraint. Then we show how it can be extended to the case ofmultiple constraints and multiple systems.

The procedure F requires tolerance level ε and target value q for the con-straint in consideration, y ≤ q′. The procedure has a monitoring statistic C(r)of the observations from the system where r is the current sampling stage.For example, Rinott [1978] and Kim and Nelson [2001] use sample meansand cumulative sums of differences in observations as C(r), respectively. Theprocedure also requires R(r; ε, h(·), S2(·)), which is a non-negative real-valuedfunction of r that takes the tolerance level ε, a non-negative valued function h,and the usual sample variance S2 of a number of observations as parameters.The function R(r; ε, h(·), S2(·)) defines a so-called continuation region (see Kimand Nelson [2001]). Sampling from a system continues as long as C(r) of thesystem stays within the continuation region. The function h is determined insuch a way that a procedure with R(r; ε, h(·), S2(·)) guarantees a prespecifiedprobability of correct selection.

The boundary (−R(r; ε, h(·), S2(·)), R(r; ε, h(·), S2(·))) defines a so-called con-tinuation region for the procedure. In our setting, F is a fully sequential pro-cedure where one basic observation is sampled at each stage and samplingcontinues as long as C(r) stays within the continuation region. Otherwise, theprocedure stops and a decision is made depending on the exit boundary. Forexample, Figure 2 shows a triangular continuation region where the horizontaland vertical axes in the figure denote stage number r and C(r), respectively.If the exit occurs through the upper boundary, we conclude that the system isinfeasible. On the other hand, if the exit occurs through the lower boundary, we



Fig. 3. Algorithmic statement of F .

conclude that the system is feasible. This continuation region is set up in sucha way that the actual PCD is guaranteed to be at least 1−α through the choiceof an appropriate value of h(·). The generic algorithm of F is given in Figure 3.

To extend F to a general case of k ≥ 1 and s ≥ 1, we need F to satisfy thefollowing assumption:

ASSUMPTION 2. When there is only one system with one stochastic constraint,h(n0) can be determined such that F satisfies PCD ≥ 1 − α.

In the rest of this section, all lemmas and corollaries will be given under theassumption that Yij and F satisfy Assumptions 1 and 2.

Now we extend F to the case of k systems and s constraints. Let Ci�(r)represent the monitoring statistic of observations Yi�j for j = 1, 2, . . . , r for thefeasibility check of the �th constraint of system i; S2

i�(n0) be the usual samplevariance of the observations Yi�j for j = 1, 2, . . . , n0 from the �th constraintof system i; R(r; ε�, h(n0), S2

i�(n0)) denote a certain shaped boundary for the �thconstraint of system i; Ki be the set of constraint indices that are checked asfeasible for system i by our procedure; and CDi� and ICDi� be the correct andincorrect decision events, respectively, when the �th constraint of system i isconsidered in isolation. Then FB is given as in Figure 4.

LEMMA 1. If β = α/(ks), then FB satisfies PCD ≥ 1 − α.

PROOF. One can notice that FB is same as applying F to each constraint ofeach system in isolation with h(n0), such that Pr{CDi�} ≥ 1 − β. Then

PCD ≥ Pr{ ∩k

i=1 ∩s�=1CDi�

}≥ 1 −

k∑i=1

s∑�=1

Pr{ICDi�}

≥ 1 −k∑

i=1

s∑�=1

β

= 1 − ksβ = 1 − ksα

ks= 1 − α,

where the second inequality follows from the Bonferroni inequality.



Fig. 4. Algorithmic statement of FB.

COROLLARY 1. If each system is simulated independently without the use ofCRN across systems, then β = (1 − (1 − α)1/k)/s satisfies PCD ≥ 1 − α.

PROOF.

PCD ≥ Pr{ ∩k

i=1 ∩s�=1CDi�

}=

k∏i=1

Pr{ ∩s

�=1 CDi�}

≥k∏

i=1

(1 −

s∑�=1

Pr{ICDi�})

≥ (1 − sβ)k

= (1 − (1 − (1 − α)1/k)k = 1 − α,

where the second inequality follows from the Bonferroni inequality.

Note that (1 − (1 − α)1/k)/s is slightly larger than α/(ks), which results inslightly smaller h(n0) and thus a slightly tighter continuation region.

3.2 Accelerated Feasibility Check Procedure (FA)

When the number of systems increases, FB becomes conservative mainly dueto the Bonferroni inequality. To lessen this conservatism, we develop a screen-ing procedure in which basic observations across constraints are aggregatedinto one observation by a linear combination. More specifically, an aggregatedobservation Y a

ij is defined as Y aij = aT Yij, where a = (a1, a2, . . . , as)T is a vec-

tor of any nonnegative real-valued constants. The choice of the vector a isdiscussed in the Appendix. Since Yij, j = 1, 2, . . . , are assumed to be IID mul-tivariate normal, the aggregated observations Y a

ij , j = 1, 2, . . . , are also IID



Fig. 5. Da, Aa, and Ua regions for the aggregated measure when there are two stochastic con-straints.

normal. Therefore, we can apply FB directly to the aggregated observationswith the aggregated tolerance level εa = aT e and the aggregated target valueqa = aT q.

The difficulty is that aggregation results in different desirable, acceptable,and unacceptable regions from those defined by the original individual con-straints. More specifically, let Da, Aa, and Ua denote desirable, acceptable, andunacceptable regions defined by aggregation with a vector a. The shaded trian-gles of Figure 5 show the three regions for the aggregated constraint with Y a

ij ,εa, and qa while rectangles show those for the original individual constraintswhen there are two constraints. As one can see from Figure 5, the systems inUa and Aa fall into the unacceptable region U or acceptable region A in termsof the original constraints, so the screening procedure based on aggregatedobservations is likely to eliminate systems in U and A only. However, Da andAa contain some unacceptable systems in terms of the original constraints.Therefore, it is possible that a system declared feasible by the screening pro-cedure based on aggregated observations is actually an unacceptable systemin U. This implies that we can confidently eliminate a system if the system isdeclared infeasible by the screening procedure based on aggregated observa-tions, but a decision that a system is feasible by the screening procedure is notsufficient to eliminate the system. Therefore, the screening procedure based onaggregated observations cannot be used solely to make the feasibility decision,but can help accelerate the elimination of unacceptable and some acceptablesystems.

Let S2i (n0) represent the sample variance of Y a

ij for j = 1, 2, . . . , n0; Cai (r)

be the monitoring statistic of aggregated observations Y aij for j = 1, 2, . . . , r



Fig. 6. Algorithmic statement of FA.

of system i; and Ra(r; εa, h(n0), S2i (n0)) be the continuation boundary for the

aggregated constraint of system i. Moreover, define CDai as the event for which

a procedure based on aggregated observations Y aij declares system i in U a to be

infeasible and system i in Da to be feasible. The accelerated feasibility checkprocedure called FA combines the screening procedure based on aggregatedobservations with FB, and is given in Figure 6.

LEMMA 2. If one chooses β0 and β1 such that kβ0 +ksβ1 = α, then FA satisfiesPCD ≥ 1 − α.

PROOF.

PCD ≥ Pr{( ∩k

i=1 ∩s�=1CDi�

) ∩ ( ∩ki=1 CDa

i

)}= Pr

{ ∩ki=1 ∩s

�=1CDi�} + Pr

{ ∩ki=1 CDa

i

}−Pr{( ∩k

i=1 ∩s�=1CDi�

) ∪ ( ∩ki=1 CDa

i

)}≥ Pr

{ ∩ki=1 ∩s

�=1CDi�} + Pr

{ ∩ki=1 CDa

i

} − 1

≥ (1 − ksβ1) + (1 − kβ0) − 1 = 1 − α,

where the third inequality follows from the Bonferroni inequality and the sec-ond equality follows from the way we choose β0 and β1.

COROLLARY 2. If systems are simulated independently without CRN acrosssystems, and β0 and β1 are chosen such that (1 − β0)k + (1 − sβ1)k = 2 − α, thenFA satisfies PCD ≥ 1 − α.



PROOF.

PCD ≥ Pr{( ∩k

i=1 ∩s�=1CDi�

) ∩ ( ∩ki=1 CDa

i

)}≥ Pr

{ ∩ki=1 ∩s

�=1CDi�} + Pr

{ ∩ki=1 CDa

i

} − 1

=k∏

i=1

Pr{ ∩s

�=1 CDi�} +

k∏i=1

Pr{CDa

i

} − 1

≥k∏

i=1

(1 −

s∑�=1

Pr{ICDi�})

+k∏

i=1

Pr{CDa

i

} − 1

≥ (1 − sβ1)k + (1 − β0)k − 1 = 1 − α,

where the third inequality comes from the Bonferroni inequality and the secondequality follows from the way we choose β0 and β1.

Lemma 2 and Corollary 2 imply that one can split the overall error α be-tween screening procedures based on original observations Yi�j and aggregatedobservations Y a

ij . This is similar to the decomposition lemma of Nelson et al.[2001].

Remark 1. The choices of β0 and β1 affect the performance of feasibilitycheck procedures as they change the continuation region. A smaller value of β0

or β1 leads to a wider (thus looser) continuation region. If β1 is too small, thenit may take long to find the feasibility of a system when it is not eliminated byscreening based on aggregation. If β0 is too small, then elimination by aggre-gation may not be effective. To keep a balance between them, we recommendtaking β0 = β1.

3.3 Accelerated Feasibility Check Procedure with Variance Update (FA+)

To further improve the performance of FA, we design a variance-updatingversion of FA, which is called FA

+. The FA+ procedure is basically same as

FA except that we update variance estimates as more observations becomeavailable. The generic algorithm of FA

+ is given in Figure 7.For a finite sample size FA

+ is heuristic, but Malone et al. [2005] show thatthe degradation from the nominal confidence level 1 − α is insignificant for thefinding-the-best problem when a variance-updating version of a statisticallyvalid procedure for IID normal observations is applied to IID normal observa-tions. Therefore, we expect the degradation from the nominal PCD in FA

+ tobe insignificant as well.

4. EXAMPLE PROCEDURES

In this section, we construct example procedures of FB, FA, and FA+. The key

in constructing a statistically valid procedure is to know how to determine acontinuation region so that the probability requirement is satisfied. The con-tinuation region can be triangular or parabolic. In Kim and Nelson [2001], atriangular continuation region is used for the finding-the-best problem. Baturand Kim [2006] show that a parabolic continuation region can be more efficient



Fig. 7. Algorithmic statement of FA+.

than a triangular continuation region in terms of the total number of observa-tions in some cases. As long as a continuation region that works for the case ofone system and one constraint can be determined, it can be extended to moregeneral cases using Lemmas 1 and 2. As (1) the goal of this section is to illus-trate an example of extending a procedure from one system and one constraintto multiple systems and constraints, and (2) a feasibility check procedure witha triangular region is already presented in Andradottir et al. [2005] and An-dradottir and Kim [2010], we focus on example procedures with a triangularcontinuation region.

Andradottir et al. [2005] and Andradottir and Kim [2010] present aprocedure—called Algorithm I—that checks the feasibility of k systems in thepresence of one constraint, satisfying the PCD requirement. By setting k = 1,we take their Algorithm I as our basic procedure F I and extend it to a moregeneral case to get F I

B, F IA, and FI+

A . Before presenting extended proceduresfor multiple constraints, we need to define some notation:

g(η, d) ≡c∑

�=1

(−1)�+1(

1 − 12I(� = c)

) (1 + 2η(2c − �)�

c

)−d/2

and

R(r; v,w, z) ≡ max{0,

wzv

− v

2cr}

,

where I is the indicator function, c is any positive integer, v,w, z ∈ R andv,w, z > 0.

Remark. A popular choice for c is 1 (see Kim and Nelson [2001]).

The procedure F IB is described in Figure 8.



Fig. 8. Algorithmic statement of F IB.

THEOREM 1. If the parameter β is set to β = α/(ks) when CRN is employedor β = (1 − (1 − α)1/k)/s when systems are simulated independently, then the F I

Bprocedure guarantees PCD ≥ 1 − α.

PROOF. Andradottir and Kim [2010] prove Pr{CDi�} ≥ 1 − g(η, n0 − 1). Sincewe set g(η, n0 − 1) = β, Pr{CDi�} ≥ 1 − β. Then by Lemma 1 and Corollary 1, itis straightforward to show that PCD ≥ 1 − α.

The procedure F IA, the accelerated version of F I

B, is given in Figure 9.

THEOREM 2. Suppose that the parameters β0 and β1 are set to β0 = β1 = γ ,where γ = α/(k(s + 1)) when CRN is employed or γ satisfies (1 − γ )k + (1 −sγ )k = 2 − α when systems are simulated independently. Then F I

A guaranteesPCD ≥ 1 − α.

PROOF. Andradottir and Kim [2010] prove Pr{CDi�} ≥ 1 − g(η, n0 − 1) = 1 −β1 = 1−γ , where the equality holds due to the way we choose η. One can noticethat the screening procedure with aggregated observations is basically, sameas Algorithm I of Andradottir and Kim [2010], with εa, qa, and Y a

ij . ThereforePr{CDa

i } ≥ 1 − g(η, n0 − 1) = 1 − β0 = 1 − γ , where the equality holds by the



Fig. 9. Algorithmic statement of F IA.

way we choose η. Also, note that kγ + ksγ = α and (1 − γ )k + (1 − sγ )k = 2 − α.Then by Lemma 2 and Corollary 2, the theorem follows.

As shown in Theorem 2, F IA guarantees CD with probability at least 1 − α.

Choosing β0 = β1 = γ = α/(k(s + 1)) guarantees CD with probability at least1 − α. However, with this choice of β0 and β1, F I

A might not always per-form better than F I

B. For example, suppose that there are two systems withtwo constraints with the overall nominal confidence level 95% and we chooseβ0 = β1 = 0.05/6 ≈ 0.0083. If all the systems are in the desirable region D,elimination based on aggregated observations is unlikely to be utilized. ThenF I

A becomes very similar to F IB except that η of F I

A is based on the larger con-fidence level, 1 − β1 = 0.9917, instead of 1 − β = 1 − 0.05/4 = 0.9875 of F I

B.Thus, the performance of F I

A is likely to be worse than F IB under this situation.

On the other hand, it is possible that F IA performs better than F I

B if thereare a number of acceptable or unacceptable systems in terms of the originalconstraints. More specifically, the savings from eliminating some acceptable orunacceptable systems earlier by screening based on aggregated observationsmight be large enough to compensate a larger probability 1 − β1 than 1 − β.Similar arguments apply when systems are simulated independently.

To avoid this possible inferior performance of F IA to F I

B, one may choose touse γ = α/(ks) for CRN and γ = (1 − (1 − α)1/k)/s for the independent casein F I

A, which are the same values of β in F IB. This way F I

A is guaranteed toperform better than F I

B in terms of the number of observations required untilwe reach a decision. Unfortunately, with this choice of γ , PCD for F I

A is nowonly guaranteed to be ≥ 1 − α − α/s for CRN and ≥ {1 − (1 − (1 − α)1/k)/s}k − α

for the independent case. However, it will certainly guarantee F IA to perform

as good as or better than F IB under any circumstances while the actual PCD is



Fig. 10. Algorithmic statement of FI+A .

most likely to satisfy the PCD requirement because α/s and (1−(1−α)1/k)/s aresmall and the procedures tend to be conservative for large k due to the use ofthe Bonferroni inequality and the nature of indifference-zone type procedures(see Kim and Nelson [2006b]).

Finally, the variance-updating version FI+A is given in Figure 10. As dis-

cussed in Section 3.3, FI+A is heuristic.

5. EMPIRICAL EVALUATION

In this section, we compare the performance of F IB, F I

A, and FI+A .

5.1 Multivariate Normal Example

Output vectors corresponding to s constraints for system i are directly gen-erated from a multivariate normal distribution with a mean vector yi and apositive definite variance-covariance matrix �i.

We consider three mean configurations yi from each of desirable, acceptable,and unacceptable regions. The mean configuration of a system in the desirableregion will take one of D1, D2, and D3 configurations. Similarly, A1 to A3are for systems in the acceptable region, and U1 to U3 are for systems inthe unacceptable region. Without loss of generality we can assume that q =(0, 0, . . . , 0)T . The tolerance level for each constraint is set to ε� = 1/

√n0. The

nine mean configurations with these choices of q and e are shown in Table I.The D1, A1, and U1 configurations are respectively more difficult than the D2,A2, and U2 configurations, and D2, A2, and U2 are more difficult than D3, A3,and U3. For example, a system with the D1 configuration barely falls into thedesirable region with yi, whose elements are exactly equal to −ε. On the otherhand, the D2 and D3 configurations have much smaller expected performancemeasures than −ε for all constraints; thus, it should be easier to detect thatthey are desirable systems.

The variance-covariance matrix of system i, �i, is assumed to have diago-nal elements σ 2

� for � = 1, 2, . . . , s and non-diagonal elements ρσ�σν for � �= ν



Table I. The Mean Configurations of thePerformance Measures Associated With s

Stochastic Constraints

desirableD1 y� = −ε, � = 1, 2, . . . , sD2 y� = −�ε, � = 1, 2, . . . , sD3 y� = −10ε, � = 1, 2, . . . , s

acceptableA1 y1 = y2 = −2ε, y� = −ε/2, � = 3, 4, . . . , sA2 y� = 0, � = 1, 2, . . . , sA3 y� = ε/2, � = 1, 2, . . . , s

unacceptableU1 y1 = y2 = −2ε, y� = ε, � = 3, 4, . . . , sU2 y� = ε, � = 1, 2, . . . , sU3 y� = �ε, � = 1, 2, . . . , s

and �, ν = 1, 2, . . . , s. That is, system i generates Yij, j = 1, 2, . . . , that aremultivariate normally distributed with mean vectors yi and marginal vari-ances σ 2

� for � = 1, 2, . . . , s and equal correlation, ρ, between each pair of con-straints. The correlation ρ varies over ρ = {−0.25,−0.15, 0.0, 0.3, 0.7}, wherethe numbers are chosen to ensure that �i is positive definite. In this article, wepresent experimental results with ρ = −0.25, 0, and 0.7 only due to space con-straints. The marginal variances σ 2

� take one of three configurations: constantvariances (CV), increasing variances (IV), and decreasing variances (DV). Thevariances in the CV configuration are all set to one. In IV and DV, the vari-ance of each constraint � = 1, 2, . . . , s, is set to 1 + (� − 1)ε and 1 + (s − �)ε,respectively.

The number of systems is k = 9, and the number of constraints is s = 5.We make 10,000 experiments (complete repetitions) and report the estimatedPCD and sample average of the total number of replications (SAR), assumingthat Yi1 j, Yi2 j, . . . , Yisj for s constraints (that is, vector Yij) are simultaneouslyobtained from a replication.

The overall nominal confidence is 1 − α = 0.95.

5.1.1 Validity. To test the statistical validity of the proposed procedures,we consider k = 1 and D1 and U2 configurations, which resemble the slippageconfiguration for many indifference-zone type procedures (see Kim and Nelson[2006b]). Table II shows the estimated SAR and PCD of F I

B and F IA when k = 1

and s = 5 with various correlations (ρ = −0.25, 0, 0.7) across constraints.The estimated PCD were all over the nominal value regardless of correla-

tions. The U2 configuration had higher estimated PCD than the D1 configu-ration. This is expected because all constraints need to be correctly detectedas feasible in D1 while just one constraint needs to be correctly detected asinfeasible in U2.

5.1.2 Efficiency. Now, we consider k = 9 systems, one from each meanconfiguration D1 through U3. Variances of the nine systems are all assumed tofollow one of CV, IV, and DV configurations. As discussed in Section 3.2, we setβ0 = β1 = γ , where γ is either α/(k(s + 1)) or α/(ks).



Table II. Sample Average of Total Number of Replications (SAR) and Estimated PCDin Parentheses for k = 1, s = 5, ρ = −0.25, 0.0, 0.7, β = α

ks for F IB, and β0 = β1 = α

k(s+1)

for F IA

CV IV DVF I

B F IA F I

B F IA F I

B F IA

ρ = −0.25 D1 72 77 126 135 125 134(0.961) (0.964) (0.956) (0.966) (0.954) (0.966)

U2 20 10 27 10 28 10(1.000) (1.000) (1.000) (1.000) (1.000) (1.000)

ρ = 0.0 D1 72 77 125 132 124 133(0.963) (0.966) (0.960) (0.960) (0.960) (0.960)

U2 20 12 28 16 28 16(1.000) (1.000) (1.000) (1.000) (1.000) (1.000)

ρ = 0.7 D1 61 65 110 117 110 117(0.968) (0.972) (0.969) (0.970) (0.967) (0.970)

U2 27 28 36 38 36 38(1.000) (1.000) (1.000) (1.000) (1.000) (1.000)

Table III. Sample Average of Total Number of Replications (SAR) for Each System and EstimatedPCD When k = 9, s = 5, ρ = 0.0, β = α

ks for F IB, and β0 = β1 = γ for F I

A. On the Bottom, theUpdating Row Shows SAR for FI+

A and its Estimated PCD are in the Last Row

CV IV DVF I

B F IA F I

B F IA F I

B F IA

γα

k(s + 1)α

ksα

k(s + 1)α

ksα

k(s + 1)α

ksD1 145 153 145 256 268 255 257 270 254D2 96 101 95 112 117 111 208 219 207D3 20 21 20 36 38 36 36 38 36A1 207 215 200 363 376 352 363 375 355A2 128 79 76 201 125 116 201 124 115A3 61 28 27 88 44 41 87 44 41U1 53 51 49 98 96 90 65 65 61U2 43 19 18 60 30 28 60 30 28U3 18 11 11 32 14 14 21 14 13Total 771 678 641 1246 1108 1043 1298 1179 1110PCD 0.994 0.994 0.994 0.994 0.994 0.993 0.995 0.994 0.994Updating 333 322 508 490 541 523PCD 0.991 0.991 0.991 0.990 0.992 0.991

The results with different correlations are shown in Tables III–V. For ρ = 0,as shown in Table III, statistically valid F I

A with γ = α/(k(s + 1)) spendsabout 9% ∼ 12% fewer total number of replications than F I

B. Clearly, F IA with

γ = α/(k(s + 1)) spends more observations for desirable systems but savingsfrom eliminating some of acceptable and unacceptable systems earlier are largeenough to compensate it. Thus, the F I

A procedure is useful when faced with alarge number of systems, some of which are in acceptable and unacceptableregions. When γ = α/(ks), F I

A spends the same number of replications for de-sirable systems (except insignificant differences due to randomness) but fewerreplications for acceptable and unacceptable systems compared to F I

B. Thisgives around 14% ∼ 17% savings in total number of replications compared toF I

B.



Table IV. Sample Average of Total Number of Replications (SAR) for Each System andEstimated PCD When k = 9, s = 5, ρ = −0.25, β = α


A

CV IV DVF I

B F IA F I

B F IA F I

B F IA

γα

k(s + 1)α

ksα

k(s + 1)α

ksα

k(s + 1)α

ksD1 144 151 144 255 268 254 254 270 255D2 96 101 96 111 116 111 209 220 207D3 20 21 20 35 38 35 35 37 35A1 207 222 210 368 387 367 369 388 364A2 116 31 30 177 32 31 179 32 30A3 61 10 10 87 10 10 86 10 10U1 53 56 53 99 103 97 65 67 64U2 42 10 10 60 10 10 60 10 10U3 18 10 10 32 10 10 21 10 10Total 757 612 583 1224 974 925 1278 1044 985PCD 0.993 0.997 0.994 0.995 0.996 0.994 0.994 0.996 0.993

Table V. Sample Average of Total Number of Replications (SAR) for Each System and EstimatedPCD When k = 9, s = 5, ρ = 0.7, β = α


A

CV IV DVF I

B F IA F I

B F IA F I

B F IA

γα

k(s + 1)α

ksα

k(s + 1)α

ksα

k(s + 1)α

ksD1 125 132 125 229 241 229 229 242 228D2 91 96 90 100 105 100 199 212 201D3 18 19 18 33 34 33 33 34 33A1 182 191 180 329 348 326 328 345 330A2 167 172 163 269 267 252 266 271 256A3 82 84 80 111 116 110 111 116 111U1 63 67 63 118 124 118 77 80 76U2 55 57 55 74 77 73 75 78 74U3 21 22 21 40 40 38 23 24 23Total 804 840 795 1303 1352 1279 1341 1402 1332PCD 0.996 0.996 0.995 0.996 0.995 0.996 0.995 0.995 0.994

The statistically valid F IA procedure performs even better when ρ = −0.25,

as shown in Table IV. The F IA procedures with γ = α/(k(s + 1)) and α/(ks)

give around 19% and 23% savings, respectively. However, when ρ = 0.7 as inTable V, F I

A with γ = α/(k(s + 1)) spends about 4% more replications thanF I

B, while the choice of γ = α/(ks) gives a slightly smaller savings of about1%. The efficiency of FA depends on correlation because the variance of ag-gregated observations affects the continuation region. The continuation regionis larger for positively correlated observations and smaller for negatively cor-related observations. The experimental results in Table V should be consid-ered as an extreme as it is very rare that all constraints are highly positivelycorrelated.

Overall, the choice of γ = α/(ks) seems the best in practice as it always guar-antees better performance of FA over FB while getting the benefit of aggrega-tion for most cases (except highly correlated secondary performance measures).



Also, it seems to satisfy the PCD constraint in most cases as in Tables III–V,where the estimated PCD are all well over 0.95, although F I

A is only guaranteedto deliver a PCD of at least 0.94 theoretically.

The variance-updating version, FI+A , further increases the efficiency of F I

Aalmost by 58%. For example, when ρ = 0 with the CV configuration, FI+

A withγ = α/(k(s+1)) and γ = α/(ks) spend only 333 and 322 replications, respectively,with the estimated PCD of 0.991 for both cases.

We recommend the FI+A procedure with γ = α/(ks) as it is the most efficient

feasibility check procedure, with a good estimated PCD based on the empiricalresults. However, if statistical validity is important, the F I

A procedure withγ = α/(k(s + 1)) is recommended.

5.2 Robustness

So far we considered constraints on expectation. However, one may be inter-ested in a constraint on a probability or variance. Note that a probability expres-sion related to any random variable can be expressed as the expectation of anindicator random variable that takes only 0–1 values: Pr(X < a) = E[I(X < a)].Hence, any probability constraint can be written as the expectation of aBernoulli random variable.

A variance constraint for IID observations can also be expressed as anexpectation constraint. The variance of an IID random variable is equivalentto the expectation of the usual sample variance S2: Var(X) = E[S2]. If IIDobservations are divided into batches and sample variance is calculated fromeach batch, then those sample variances become basic observations in ourprocedures.

Since the probability and variance constraints can both be expressed asexpectation constraints, our procedures work for them. However, neither theindicator random variable nor the sample variance satisfies the normality as-sumption. In Kim and Nelson [2001] and Malone et al. [2005], it is shownthat fully sequential procedures for the finding-the-best problem are robust tonon-normality and the method of batching is helpful in achieving approximatenormality. In this section, we test the robustness of the proposed procedures tonon-normality with and without batching based on IID Bernoulli distributedobservations.

We assume that there is one system (k = 1) with two constraints (s = 2)and ρ = 0, and its mean configuration takes one of the following three configu-rations:

yi

D1′ (q1 − ε, q2 − ε)U1′ (q1 − ε, q2 + ε)U2′ (q1 + ε, q2 + ε)

The first configuration is for a desirable system and the other two are forunacceptable systems. We do not consider acceptable systems because theirPCD is always one by definition. Having one system in D1′ is a difficult casein terms of PCD because all yi are exactly ε apart from q. Although a system



Table VI. Sample Average of Total Number of Replications (SAR) andEstimated PCD When Observations are Bernoulli Distributed and One

System (k = 1) is Considered with s = 2, ρ = 0.0, β = αks for F I

B, andβ0 = β1 = γ for F I

A

Batch F IB F I

A with γ = α

k(s + 1)F I

A with γ = α

ksSize SAR PCD SAR PCD SAR PCD1 D1′ 916 0.217 1039 0.205 915 0.217

U1′ 502 1.000 574 1.000 497 1.000U2′ 316 1.000 226 1.000 193 1.000

5 D1′ 2014 0.846 2402 0.852 1997 0.835U1′ 1041 0.988 1134 0.994 957 0.990U2′ 612 1.000 526 1.000 438 1.000

10 D1′ 2047 0.901 2444 0.910 2012 0.876U1′ 1084 0.989 1172 0.994 970 0.992U2′ 654 1.000 567 1.000 469 1.000

30 D1′ 2064 0.926 2472 0.937 2046 0.909U1′ 1137 0.989 1233 0.995 1035 0.991U2′ 738 1.000 639 1.000 549 1.000D1′ 2180 0.955 2560 0.958 2170 0.947

100 U1′ 1400 0.992 1490 0.995 1340 0.993U2′ 1110 1.000 1070 1.000 1040 1.000

in U2′ also has means exactly ε apart from q, this configuration is easier thanD1′, since a system in U2′ will be eliminated when at least one constraint isdetected correctly. On the other hand, a system in D1′ requires feasibility ofboth constraints to be detected correctly.

If yi are close to 0.5 or ε is large, problems become easy. To avoid it, we setq1 = q2 = 0.95 and ε1 = ε2 = 0.01. The first stage-sample size is n0 = 10 and10,000 macro-replications were made.

Table VI shows that estimated PCD can be smaller than the nominal value0.95 for the D1′ configuration with a small batch size. Without batching, esti-mated PCD for a system in D1′ are extremely small, around 0.2. However, forbatch sizes of 2 and 3, estimated PCD increases to 0.5 and 0.7, respectively(these results are not reported in Table VI due to limited space). For a batchsize ≥ 5, estimated PCD are close to 0.9, not significantly smaller than thenominal level 0.95. As the mean configuration is deviated from the D1′ con-figuration, the estimated PCD gets well over 0.95 for any batch size. As thenumber of systems or constraints increases, actual PCD tends to increase aswell due to the Bonferroni inequality (see Kim and Nelson [2006b] and Maloneet al. [2005]).

Note that this example is an extremely difficult case due to high q and yi witha small ε. When q1 = q2 = 0.7, estimated PCD for a system in D1′ is over 0.9without batching. Estimated PCD for IID exponential observations are also allover 0.9 without batching. These results are omitted from the article to conservespace. From numerical results, we conclude that our procedures are robust tonon-normality, except for extremely difficult Bernoulli cases. When the problemis difficult with extreme non-normality such as the Bernoulli observations withvery small or large q and yi, batching with a small batch size helps.



Fig. 11. Job shop example.

5.3 Queueing Example

In this section, we consider a small queueing example to evaluate the perfor-mance of F I

A, F IB, and FI+

A . There is a small job shop with nine agile workersand four stations named Stations A, B, C, and D. The diagram of this job shopis shown in Figure 11. A study shows that 30% of the items arriving at StationA move along Station B → Station C → Station D. The rest of the items (70%of them) move along Station B → Station D. Moreover, Station B has externalarrivals that move along Station C → Station D and Station C has externalarrivals that move directly to Station D.

We want to determine feasible allocations of the nine agile workers overfour stations when there are s ≤ 4 constraints that the expected steady-statenumber of jobs waiting in queue at each station � = 1, 2, . . . , s should be smallerthan or equal to a predetermined constant due to limited buffer space for thestation. We assume that the number of external arrivals to Station � followsa Poisson distribution with rate γ� and that service times of each worker atStation � are exponentially distributed with rate μ� for � = 1, 2, 3, 4. The valuesof γ� and μ� for each station � are given as follows.

Station γ� (jobs/hr) μ� (jobs/worker/hr)1 50 272 2 553 14 314 0 80

We assume that at least one worker should be assigned to each station.Then there are 56 different ways to allocate the nine agile workers over thefour stations. Among those configurations, 35 of them are stable—the expectednumber of jobs waiting in queue does not blow up in steady-state. In thisexperiment, we consider 33 of the stable configurations (thus, k = 33). Thepredetermined constant for the queue size (q�) and tolerance level (ε�) for eachstation are given in the following.

Station (�) q� (jobs) ε� (jobs)1 11.5 0.52 1 0.13 1 0.14 5 0.1



Since all service times and external interarrival times are assumed to beexponentially distributed, the expected steady-state number of jobs waiting inthe queue of each station can be analytically obtained. This enables us to checkif a tested procedure makes a correct decision on the feasibility of a system.

The observations for each system are sampled with the replication/deletionapproach with independent replications (no CRN). The warm-up period and therun length are set to 200 hours and 2360 hours, respectively. The time-averageof the queue length over the last 2160 hours in each independent replication isused as an observation from the simulated system.

Based on the findings from Section 5.1, we first tested the case of β0 = β1 =α/(ks) for F I

A and FI+A . The first-stage sample size is n0 = 10 and the nominal

confidence level is 1−α = 0.95. For k = 33 systems and s number of constraints,SAR from 100 iterations are shown in the following table.

F IB F I

A FI+A

γ = α

k(s + 1)γ = α

ksγ = α

k(s + 1)γ = α

kss = 2 1022 957 878 521 513s = 3 1793 1090 1022 486 478s = 4 1981 1097 1063 619 604

All three procedures correctly returned a set F, with 100% correctness con-taining all the desirable systems and some acceptable systems only.

For a given k and α, a large s implies a smaller γ and a wider continuationregion. Thus, one would expect the proposed procedures to spend more repli-cations as the number of constraints, s, increases. Procedures F I

B and F IA show

this tendency in the preceding table. However, when s changes, the numbers ofunacceptable, acceptable, and desirable systems also change. More specifically,among 33 configurations, there are 15 unacceptable, 10 acceptable, and 8 de-sirable systems for s = 2; and 23 unacceptable, 6 acceptable, and 4 desirablesystems for s = 3 or 4. The change in system feasibility affects the numberof replications needed until a decision is made. We believe that it partiallyexplains why FI+

A spends more replications for s = 2 than s = 3.The table also confirms that F I

A and FI+A achieve meaningful savings in SAR

compared to F IB. When s = 4, F I

B spent 1981 replications whileF IA and FI+

A withγ = α/(k(s + 1)) spent 1097 and 619 replications, respectively. This represents44.5% and 68.8% savings compared to F I

B. With the choice of γ = α/(ks), F IA

and FI+A spent 1063 and 604 replications, which represents 46.3% and 69.5%

savings compared to F IB, respectively.

6. CONCLUSION

We present three generic procedures for the purpose of identifying a set offeasible or near-feasible systems. Three example procedures with a triangular-shaped continuation region are proposed in this article. The FB procedurechecks the feasibility of k systems in the presence of s stochastic constraintswith a prespecified probability of correct decision. The performance of FB canbe accelerated by a procedure called FA, which reuses observations across



Fig. 12. Determining line �1.

constraints by applying screening to the aggregated observations. Varianceupdating further improves the efficiency of FA significantly.

The proposed procedures can solve an optimization problem where the ob-jective function is deterministic. They also serve as critical steps to solving anoptimization problem where both the objective function and the constraintsneed to be estimated by simulation.

APPENDIX

In this appendix, we propose a choice of vector a based on a simple heuristicargument. Without loss of generality, we can assume that q1 = q2 = 0 and startwith s = 2. In Figure 12, consider a line that passes through point (ε1, ε2) andhas the following form: a1x + a2y + c = 0 such that a1ε1 + a2ε2 + c = 0, a1 > 0,and a2 > 0.

Note that a region over the line corresponds to Ua and systems in the lightand dark shaded regions are infeasible systems but fall in either the Aa orDa region. Thus, those infeasible systems are not likely to be benefited fromaggregation. Hoping to minimize the possible number of such systems, one maywant to minimize the total area of the light and dark shaded regions in thefigure. However, the light shaded regions contain systems that are feasible interms of one constraint and infeasible in terms of the other constraint. In thiscase, it is not clear if aggregated observations would make it easier to detectinfeasibility. On the other hand, the dark shaded regions contain systems thatare infeasible for both constraints so aggregation likely makes infeasibility



more noticeable. Therefore, the regions that actually matter for the efficiencyof the screening part based on aggregation are the dark shaded regions. Wetherefore want to minimize the area of the dark shaded regions. As the areaof Rectangle ABCD does not depend on the choice of a1 and a2, minimizing thearea of the dark shaded region is equivalent to minimizing the area of TriangleFBE, which is c2/a1a2.

By similar arguments, for a general s, we solve

mincs

a1a2 · · · as

s.t. a1ε1 + a2ε2 + · · · + asεs + c = 0

ai > 0, i = 1, 2, . . . , s,

and get a = [a�]�=1,2,...,s such that a� = ∏sν=1,ν �=� εν .

ACKNOWLEDGMENTS

We thank Dr. Barry L. Nelson, Dr. James R. Wilson, and the associate editorand referees for their helpful comments.

REFERENCES

ANDRADOTTIR, S., GOLDSMAN, D., AND KIM, S.-H. 2005. Fully sequential procedures for comparingconstrained systems via simulation. In Proceedings of the Winter Simulation Conference. M. E.Kuhl, N. M. Steiger, F. B. Armstrong, and J. A. Joine Eds., Institute of Electrical and ElectronicsEngineers, Piscataway, NJ, 732–738.

ANDRADOTTIR, S. AND KIM, S.-H. 2010. Fully sequential procedures for comparing constrainedsystems via simulation. Naval Res. Logistics 57, 403–421.

BATUR, D. AND KIM, S.-H. 2005. Finding the set of feasible systems when the number of systemsor constraints is large. In Proceedings of the Winter Simulation Conference. M. E. Kuhl, N. M.Steiger, F. B. Armstrong, and J. A. Joines Eds., Institute of Electrical and Electronics Engineers,Piscataway, NJ, 692–698.

BATUR, D. AND KIM, S.-H. 2006. Fully sequential selection procedures with parabolic boundary.IIE Trans. 38, 749–764.

BUTLER, J., MORRICE, D. J., AND MULLARKEY, P. W. 2001. A multiple attribute utility theory approachto ranking and selection. Manag. Sci. 47, 800–816.

GOLDSMAN, D., KIM, S.-H., AND NELSON, B. L. 2005. Statistical selection of the best system. InProceedings of the Winter Simulation Conference. M. E. Kuhl, N. M. Steiger, F. B. Armstrong,and J. A. Joines Eds., Institute of Electrical and Electronics Engineers, Piscataway, NJ, 178–187.

KIM, S.-H. AND NELSON, B. L. 2001. A fully sequential procedure for indifference-zone selectionin simulation. ACM Trans. Model. Comput. Simul. 11, 3, 251–273.

KIM, S.-H. AND NELSON, B. L. 2006a. On the asymptotic validity of fully sequential selectionprocedures for steady-state simulation. Oper. Res. 54, 475–488.

KIM, S.-H. AND NELSON, B. L. 2006b. Selecting the best. In Handbooks in Operations Researchand Management Science:Simulation, S. G. Henderson and B. L. Nelson Eds., Elsevier Science,Oxford, UK, 501–534.

MALONE, G. J., KIM, S.-H., GOLDSMAN, D., AND BATUR, D. 2005. Performance of variance updatingprocedures on various data. In Proceedings of the Winter Simulation Conference. M. E. Kuhl,N. M. Steiger, F. B. Armstrong, and J. A. Joines Eds., Institute of Electrical and ElectronicsEngineers, Piscataway, NJ, 825–832.

NELSON, B. L., SWANN, J., GOLDSMAN, D., AND SONG, W. 2001. Simple procedures for selecting thebest simulated system when the number of alternatives is large. Oper. Res. 49, 950–963.



RINOTT, Y. 1978. On two-stage selection procedures and related probability-inequalities. Comm.Stat.-Thy. and Meth A7, 799–811.

SANTNER, T. AND TAMHANE, A. 1984. Designing experiments for selecting a normal population witha large mean and a small variance. In Design of Experiments—Ranking and Selection: Essays inHonor of Robert E. Bechhofer. Marcel-Dekker, New York, 179–198.

Received February 2007; revised February 2008, September 2008, March 2009, June 2009; accepted June 2009


finding feasible systems in the presence of constraints on multiple performance measures

Documents