Requirements Trade-offs Analysis in the Absence ofQuantitative Measures: A Heuristic Method

Golnaz ElahiDepartment of Computer Science

University of Toronto, Canada, M5S 1A4gelahi@cs.toronto.edu

Eric YuFaculty of Information

University of Toronto, Canada, M5S 3G6yu@ischool.utoronto.ca

ABSTRACTSimultaneously satisfying multiple interacting and possiblyconflicting software requirements is challenging. Quanti-tative cost-benefit analysis of alternative solutions is oftenhard or biased, and early decisions based on numerical esti-mates of requirements satisfaction are thus unreliable. Wepropose a trade-off analysis method that assists decisionmaking in the absence of numerical data. We structurethe requirements trade-off problem in terms of a goal modelcontaining alternative design solutions and decision criteria.We propose a trade-off analysis algorithm that takes pair-wise comparisons of alternatives and determines the bestsolution among alternatives. The optimum alternative isdecided by using a heuristic method, which may need to con-sult with domain experts. We take advantage of the EvenSwaps method [1] to incorporate stakeholders’ preferencesinto the decision analysis. The algorithm is implemented ina prototype tool and evaluated in an industrial case study.

1. INTRODUCTIONRequirements Engineering (RE) usually involves sacrific-

ing the implementation of some requirements for satisfy-ing some other requirements. These trade-offs among re-quirements may be originated from conflicting goals of dif-ferent stakeholders and interactions among Non-FunctionalRequirements (NFRs), because usually satisfaction of onerequirement by adopting a design solution can aid or de-tract from the satisfaction of other NFRs or obstruct somefunctionality [2].

In current practice, software professionals need to makethese trade-offs when selecting technologies being used, ar-chitectural patterns, and design solutions. These key earlydecisions in the project have long-term and critical impactson the product [3]. In the ideal case, if the (financial) costs,benefits, risks, and utility of each alternative solution wereknown and one could accurately estimate how well each de-sign solution satisfies the requirements, alternative solutionscould be objectively compared.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.SAC’11 March 21-25, 2011, TaiChung, Taiwan.Copyright 2011 ACM 978-1-4503-0113-8/11/03 ...$10.00.

Quantitative approaches can help determine the optimumsolution objectively by applying mathematical operationson numerical trade-off factors. These mathematical oper-ations are seen as important tools to support objective de-cision analysis; therefore, many of the requirements decisionmethods [3, 4, 5, 6] extensively rely on the availability, accu-racy, and meaningfulness of numerical estimations of risks,costs, benefits, and the satisfaction level of requirements.A fundamental assumption made by these methods is thatsoftware analysts and stakeholders have the cognitive abil-ities and empirical knowledge required to provide accurateand complex quantitative requirements models as well as re-quirements satisfaction evaluations in the form of absolutemeasures.

However, in practice, estimating or measuring the satis-faction level of NFRs is difficult. Stakeholders are usuallyresponsible for providing such numerical values [7], however,they are often judgmental and biased, and quantitative val-ues elicited from stakeholder are often unreliable for decisionmaking [8, 9]. Furthermore, at this stage, time and budgetlimitations preclude elaborate methods for obtaining quanti-tative data. Hence, faced with the typical absence of reliablequantitative data, trade-off decision analysis methods usingqualitative values are needed.

Goal-oriented requirements modeling and analysis ap-proaches such as the NFR [10] and i* Framework [11] enablestructuring and qualitatively evaluating system and user’srequirements. Goal models can be used to structure therequirements trade-off problems. Figure 1 shows a simplegoal modeling notation. The notation consists of two mainmodeling elements: goals and solutions. Using AND/ORlinks, high-level and subjective goals are decomposed andrefined into a deep enough hierarchy, in such a way that theleaf goals represent the criteria for trade-off decision mak-ing. The consequences of a solution on a goal are expressedby help (+) and hurt (-) links, similar to the i* notation’s“contributions” [11].

An Illustrative Example: We motivate our work with ascenario at the Ministry of Transportation, Ontario (MTO),Canada. Stakeholders at MTO must decide whether to keepan existing traffic management system or deploy a new web-service based Intelligent Transportation System (ITS). Thegoal of the ONE-ITS system is to provide and distributethe data necessary to carry out traffic management oper-ations and amalgamate all of the information sources intoone platform. Although the new system facilitates sharingand distributing traffic data, decision makers at the MTOare concerned about unknown security threats against the

651

Figure 1: The simplified goal modeling notation

web-service access to the traffic data. Managers at the MTOhave to deal with other minor and major trade-offs. For ex-ample, Figure 2 shows the goal model and trade-offs withinthe Variable Message Sign (VMS) management sub-system.

One requirement of the VMS sub-system is to Update themessages easily, which needs Easy to learn VMS manage-ment and Simple VMS manipulation toolkit. Figure 2 de-picts how two alternative systems (A1, the new ONE-ITSsystem and A2, the current centralized data provider) con-tribute to the requirements of the VMS system. The ONE-ITS system Display the VMS devices on an electronic map,which “helps” satisfy the Easy to learn VMS managementand Simple VMS manipulation toolkit. However, ONE-ITSsystem Enables operations over a web portal which “hurts”the Secure modification of messages.

Figure 2: The goal model for the VMS sub-system

In order to decide between these two alternative systems,MTO managers must examine costs (and other negative im-pacts) and benefits of the new proposal compared to theexisting situation. However, measuring those qualities forthe ONE-ITS system, which only exists as a system require-ments specification, is not feasible. Quantitatively measur-ing the security level, usability, and maintainability of theexisting system is challenging if not impossible. In order tomake a final decision, the MTO managers may need to relyon estimations of those factors for the ONE-ITS; however,estimations may reduce the confidence of the final decision.

Contributions: In section 2 and 3, we overview the ex-isting NFRs evaluation and decision analysis methods, anddiscuss the limitations and capabilities of current quantita-tive and qualitative methods. In section 4, we propose atrade-off analysis method that relies on comparing alterna-tive solutions rather than estimating the requirements sat-isfaction levels in the form of absolute or ordinal numbers.

In this method, instead of transforming the comparisons toa numerical representation of utility or preferences (the waythat Analytic Hierarchy Process (AHP) [12] works), all validsatisfaction levels that the requirements could possibly haveare enumerated, with respect to the relative rankings of al-ternatives. The algorithm determines the better alterna-tive for each of these possible goal satisfaction levels. Then,by using a heuristic method, which may need to consultwith domain experts, the overall optimum solution is deter-mined. In this step, to avoid eliciting numerical importanceweights of goals we take advantage of the Even Swaps [1] forconsidering stakeholders’ preferences in the decision makingprocess. Even Swaps is a recently introduced multi-criteriadecision analysis method which resolves trade-offs by swap-ping conflicting goals. In the rest of this paper, we use thescenario at the MTO to illustrate our method. In section 5,the result of this case study is discussed.

2. RELATED WORKMulti-Criteria Decision Analysis (MCDA) is an umbrella

term that groups a collection of formal approaches that takeinto account multiple criteria and help decision makers ex-plore decisions when intuitive, gut-feeling decision making isnot satisfactory [13]. These methods have been successfullyapplied in the fields of economics, operation research, andmanagement science, and can potentially be used in RE fordeciding on alternative solutions to satisfy requirements.

For example, the Multi Attribute Utility Theory (MAUT)(Keeney and Raiffa [14]) focuses on the use of multi attributepreference models. In MAUT, the weight and value of cri-teria i (w(i) and v(i) respectfully) are conjointly used tocalculate the total utility value, u =

∑i w(i)v(i). Prefer-

ence theory studies the fundamental aspects of individualchoice behavior, such as how to identify and quantify anindividual’s preferences over a set of alternatives, and howto construct appropriate preference representation functionsfor decision making [15].

AHP is a theory of relative measurement of intangiblecriteria to derive a scale of priorities (preferences, impor-tance weights) from pairwise comparisons of elements [12].AHP has been applied for prioritizing software requirements[16] and calculating relative weights of non-functional re-quirements [5, 4] or crosscutting concerns in aspect-orientedrequirements engineering [17]. Yen and Tiao [18] also usepairwise comparisons of criteria, but drive the priorities byMarginal Rate of Substitution [14]. The Even Swaps [1]method avoids eliciting explicit numerical priorities, and yetincorporate the stakeholders’ preferences in deciding on theoptimum solution by querying value trade-offs for specifictrade-off cases.

Trade-off analysis is the systematic examination of advan-tages and disadvantages of requirements as well as the designchoices for a system to achieve the right balance among sev-eral competing goals [19]. Architecture Tradeoff AnalysisMethod (ATAM) [20] is used to evaluate whether an archi-tecture decision satisfies particular quality goals. SecurityVerification and security solution Design Trade-off analysis(SVDT) [21] specializes for security solution design trade-off and risk assessment. It uses a Bayesian Belief Net witha fixed set of trade-off parameters to support design deci-sions. Feather et al. [3] propose a quantitative model forstrategic decision analysis and trade-off analysis consider-ing quality requirements, by the “coarse quantification” of

652

relevant risk factors and their interactions. In [18], fuzzylogic values are used for representing and reasoning on thesatisfaction degree of imprecise (quality) requirements. In[22], attribute values, such as contribution values and prefer-ence matrices, are added to goal graphs to choose and adopta goal from the alternatives and to recognize the conflictsamong the goals. Letier and Lamsweerde [7] argue that dueto the lack of accuracy and measurability of goal formula-tions and lack of impact propagation rules through the goalmodel, domain-specific quality variables are needed to rea-son on partial goal satisfaction. In [7], goals are specified ina precise probabilistic way, and the impacts of alternativedecisions on the degree of goal satisfaction is analyzed.

3. CHALLENGES OF REQUIREMENTSTRADE-OFF ANALYSIS

Various MCDA methods have been used for requirementspreference analysis and solution selection [5, 4, 16, 18]. Goalmodel evaluation techniques such as [10, 23] are also usefulfor assessing the extent to which alternative solutions satisfyquality goals. Quantifying the contributions and goal satis-faction levels is a common approach and has been adoptedin several proposals [5, 24, 25, 26, 27]. These different ap-proaches are useful for specific usage scenarios, dependingon the availability of numerical data, expertise of stakehold-ers, and the type of analysis that needs to be done. Wereview the limitations of different approaches in the contextof requirements trade-off analysis.

1. Low granularity of evaluation labels: Goal model-ing methods that rely on goal refinement for achieving finer-grained differentiation among alternatives typically employa minimal set of labels to indicate the satisfaction level ofgoals. For example, in [10, 23, 28], possible satisfaction de-grees are partially satisfied ( ), fully satisfied ( ), partiallydenied ( ), fully denied ( ), conflict ( ), and unknown ( ).These labels are meant to be used in conjunction with it-erative goal refinement to arrive at satisfactory solutions.However, such satisfaction levels do not have precise mean-ing for decision analysis purposes. Stakeholders are oftenable to make finer distinctions among a given a set of alter-natives with respect to impacts of alternatives on the goals.MCDA approaches therefore employ more granular scalesfor evaluation.

2. Unreliability of judgment about conflicts: Thepropagation rules in the goal model evaluation techniques of-ten result in undetermined satisfaction for higher level goalsdue to conflicting contributions. In some qualitative eval-uation methods [10, 23], when a number of elements haveconflicting contributions to a higher goal, human judgmentis required to combine the contributions and decide aboutthe evaluation value of the higher goal. This judgment canbe cognitively demanding, since it relies on domain knowl-edge rather than agreed rules, guidelines, and mathematicalor logical operations. Hence, the qualitative labels result-ing from the goal model evaluation may not assumed to bereliable for decision making.

3. Lack of operations on qualitative labels: Theset of partial and sufficient satisfaction (denial) labels isnot helpful for evaluating and comparing alternatives withrespect to multiple goals. For example, one cannot tellwhether solution A1, which satisfies G1 but denies G2, shouldbe preferred over solution A2 which partially satisfies G1 and

partially denies G2. ( + S + )

4. Ad-hoc treatment of preferences: Relative at-tractiveness (also known as preferences, relative importance,priorities) of goals are not usually captured in goal models,or if considered, are incorporated into the goal model evalu-ation methods by the use of ordinal importance weights. Ingeneral, eliciting requirements preferences is a challengingprocess and requires stakeholders to answer cognitively-hardqueries. Preference elicitation methods such as AHP may re-

quire up to m2

2number of queries asked from stakeholders

to compare preferences of m goals.

5. Non-linearity of value functions: Utility func-tions calculate the utility of an alternative over a set of goalsby summing up the product of value functions and weights(preference) of the goals. When assessing requirements satis-faction, the value function is not necessarily a linear functionof satisfaction level of requirements. For example, assumethe security level of a system is represented by percentageswhere a satisfaction level like 80% indicates the system’savailability and data confidentiality is guaranteed for 80%of the time. However, the security value of the 80% guar-antee is not necessarily 0.8 in the scale of 0-1. Stakeholdersmay believe if the security is not guaranteed for more than95% of the time, the security value is 0. The security levelof 95% may provide a value equal to 0.5, the value for 99%is 0.9, etc. Hence, the value of different satisfaction levels ofgoals needs to be elicited from the stakeholders, which is alabor-intensive and costly process.

Figure 3: Quantifying goal satisfaction levels

6. Quantification challenges: Quantification of con-tributions (and goal satisfaction) enables applying quantita-tive MCDA methods such as Utility Theory to find the op-timum solution; however, Letier and Lamsweerde [7] arguethat numerical goal satisfaction values may have no physicalinterpretation in the application domain. An agreed scale ofmeasurement and rigorous ways to measure alternatives im-pacts on goals are not available, and current approaches arenot clear about how the numerical data are obtained fromstakeholders. Acquiring and applying accurate and reliablequantitative information from stakeholders and domain ex-perts is challenging, because:

• Implicit and unintended rankings: These num-bers are not obtained by objective measurements, butinstead, decision stakeholders judge the strength ofalternatives by choosing a value between a predeter-mined lower limit for the worst possible alternativeand a predetermined upper limited for the best alter-natives [29]. Although stakeholders implicitly com-pare each alternative to a minimum and maximumcontribution strength, alternatives are not explicitlycompared with each other. For example, in Figure 3,Sat(G2, A2) > Sat(G2, A1) (Sat(g, A) denotes the sat-isfaction level of goal g by alternative A). This rankingis deduced from the ordinal values, but may have beenimplicit and invisible to the stakeholders who providedthe numbers. If stakeholders are asked to compare andrank the strength of alternatives, they may provide dif-

653

ferent numerical values for the contributions strength.

• Incorrect mathematical judgment: Numerical val-ues may actually obscure the fact that the numbersare based on estimates in an ordinal scale. Therefore,stakeholders and decision analysts may assume whenSat(G2, A1) = 7 and Sat(G2, A2) = 1, then A1 is 7times better than A2 for satisfying G2. However, thesenumerical values are rough estimations in an ordinalscale, assumptions about their ratio or intervals aremathematically incorrect, and the use of utility theoryin this context is not justifiable.

• Relativity of numerical values: If an alternativesuch as A3 is added to the model in Figure 3, andSat(G1, A3) > Sat(G1, A2), then the satisfaction levelof G1 by A3 needs to be expressed by a number greaterthan 100. However, Sat(G1, A2) is the maximum de-gree in the scale of -100 to 100; therefore, all of thegoal satisfaction levels need to be adjusted downwardsin order to fit the new alternative (A3) within the scale.

• Limited meaningful arithmetic operations: Of-ten, alternatives’ contribution strength are elicited inordinal scales [12], while limited meaningful arithmeticoperations can be applied to the ordinal scale mea-sures. The evaluation methods that calculate goal sat-isfaction levels or propagates the evaluation values inthe goal graph and calculate an ultimate utility for al-ternatives need to avoid mathematically meaninglessarithmetic such as average over the ordinal input.

7. Impractical comparisons: When quantification ofgoal satisfaction levels is costly or not possible, comparinggoal satisfaction levels from stakeholders’ point of view canprovide a rich source of information for trade-off analysis.Such information would be useful to rank the alternativesbased on the level of goal satisfaction that each alternativeprovides. For example, in Figure 4, stakeholders are moresatisfied with the security level of solution A1 than its ease ofuse. G1.sat

G2.sat= High captures this comparison in an ordinal

scale. However, comparing the satisfaction level of disparategoals such as security vs. usability is by nature hard andcognitively impractical.

Figure 4: Comparing goal satisfaction levels

8. Lack of rules for propagating comparisonsthrough the goal hierarchy: In order to avoid compar-ing the satisfaction level of disparate goal, one can comparethe contribution strength of alternatives. For example, inFigure 5, A1 is highly stronger than A2 for satisfying G1

( A11A21

= High), and A2 is somehow stronger than A1 for

satisfying G2 ( A22A12

= Medium). Comparing alternatives onthe same goal is more intuitive and natural than providingabsolute measures; however, the comparisons of contribu-tions cannot be propagated through the goal graph, becausethese comparisons lack the information about G1 and G2’ssatisfaction levels.Summary. Table 1 compares four methods for analyzing

alternative solutions with respect to multiple goals. Method

Figure 5: Comparing the contributions of alterna-tive solutions

one: quantifying alternatives and satisfaction level of goals.Method two: comparing the goals satisfaction levels. Methodthree: comparing the contributions of alternatives. Methodfour: deriving numerical values from comparison of decisionelements (AHP).

In order to quantify contribution levels in a simple two-layers goal graph, one query per contribution link is needed,hence, in total, m × n number of queries need to be askedfrom stakeholders (having n alternatives and m goals). Whencomparing the satisfaction level of goals for each alternative,every goal is compared with the rest of goals that have notbeen compared to, which requires m − 1 + m − 2 + ... + 1comparison, so in total, m×m−1

2× n queries are needed.

When comparing the contributions of a pair of alternatives,m queries for comparing a pair of alternatives are needed. Ifthe dominated alternative in the pair is eliminated from thelist of alternatives, in total by n − 1 ×m queries, the bestsolution can be found. Finally, when applying the AHP,all alternatives are compared in pairs for every goal, which

requires ( n2

2×m) number of queries.

In Table 1, the cognitive complexity for providing the in-put to these four methods is analyzed subjectively. Differ-ent methods may appear cognitively complex for differentgroups of stakeholders and in different contexts, and theevaluation given in Table 1 is not empirically and statisti-cally examined. According to the advantages and disadvan-tages of these different approaches, we use the comparisonsof contributions in our approach to analyze requirementstrade-offs. Comparing alternatives is cognitively easy andrequires the least number of queries from stakeholders.

Table 1: Different methods to increase the granu-larity of decision analysis data

Methodone

Method twoMethodthree

AHP

Number ofqueries

m× n m×m−12 ×n n− 1×m n2

2 ×m

Cognitivecomplexity

Medium Hard Easy Medium

Meaningfulpropagation

Yes No No NA

Quantitativeresults orreasoning

Yes No No Yes

4. TRADE-OFF ANALYSIS IN THEABSENCE OF NUMERICAL DATA

In this section, we present a heuristic algorithm for decid-ing over alternative solutions with respect to the trade-offsamong NFRs. In the proposed method, absolute numeri-cal measures of intangible factors such as the strength ofalternatives’ contributions or the goal satisfaction levels arenot collected. Instead, domain experts and stakeholders areasked to compare consequences of alternatives on the criteria

654

of decision making.In this method, analysts start by structuring alternative

solutions and criteria of comparisons in a simple goal modelintroduced in Section 1. The goal model refinement is re-peated until the stakeholders are able to compare the alter-natives with respect to the leaf nodes of the goal model. Thedecision analysis algorithm goes into a loop, and in each cy-cle, one pair of alternatives are compared and analyzed. Thealgorithm decides which alternative in the pair is overall abetter solution, which is called the dominant one. The infe-rior solution is called the dominated alternative. The dom-inated solution is left out from the list of alternatives andthe dominant solution is compared with the next availablealternative solution in the next cycle. The cycles continueuntil one alternative remains, which is proposed as the bestavailable design option.

In the first step of a cycle, the algorithm takes a pair ofalternatives and interacts with the users, asking them tocompare contributions of alternatives on each goal. In thesecond step, the algorithm enumerates all valid strength val-ues that the contributions of the pair of alternatives couldpossibly have. These possibilities are determined by the rel-ative orderings of contributions. In the third step, the EvenSwaps method [1] is applied to determine the dominant al-ternative for each of those possible satisfaction statuses. Inthe last step of a cycle, the overall better solution is deter-mined by applying some heuristics and consulting a domainexpert.

Step 1: Comparing the AlternativesIn the first step, stakeholders specify which alternative is

stronger for satisfying (or denying) each goals. For eachcomparison, stakeholders estimate the difference betweenstrength of the contributions. In the ONE-ITS project, weasked a traffic management expert to compare two alterna-tive systems, A1: ONE-ITS vs. A2: Individual and central-ized data providers. Both alternative systems help the Easyto learn VMS management goal, but the ONE-ITS system ismoderately better than the existing system. The MTO ex-pert described the difference of their strengths as “Medium”.This comparison is written as A1

A2= Medium for the goal

Easy to learn VMS management.

Step 2: Dealing with the Lack of Numerical Mea-sures

In this section, first we explain why comparing the alter-natives may provide enough information for determining theoptimum solution without the need for collecting numericaldata about the satisfaction level of goals. Then, using sev-eral examples, we describe how the proposed algorithm de-termines the better solution in a pair of alternative solutionsby analyzing the comparisons.

The comparisons specify how different the strength of thecontributions are, thus the algorithm is not provided withthe absolute level of contributions’ strength. For example,the domain expert may state that contributions of two al-ternatives have a small difference for satisfying a goal. Thestrength of both contributions could be extremely high orlow, or both could be medium, while in all these possibili-ties, one of the alternatives is fairly stronger than the otherone.

Since the actual contributions strength values are notknown, the algorithm cannot assign an exact number to thesatisfaction level of goals. To deal with the lack of numerical

values, the algorithm enumerates all valid contribution lev-els that the pair of alternatives could have with respect totheir relative rankings. One of these possible contributionsvalues is the actual consequences of the alternatives. In Step3, the dominant alternative is determined for each of thesepossibilities. In Step 4, the algorithm examines whether analternative is dominant for all of majority of these possiblecontribution values, and if so that alternative is probablythe better one in the pair.

One of such possible ways to place the contribution valuesof a pair of alternatives on a scale is called a “placement”case. Each possible contribution value results in a differ-ent goals satisfaction level. For example, in the ONE-ITSproject, if the difference between the contributions of A1 andA2 to the Easy to learn goal (g) is medium, and the goalsatisfaction levels are limited to 3 degrees of zero, medium,and high, there are only two possible placement cases forthese alternatives on goal g:

1) Sat(g, A1)=high and Sat(g, A2) =medium Or2) Sat(g, A1) =medium and Sat(g, A2) =zero

One of these satisfaction levels is the actual consequences ofthe A1 and A2. In both cases, Sat(g, A1) − Sat(g, A1) =medium.

The Scale of Placement Cases. We use the scaleof 0,±Low,±Medium Low, ±Medium, ±Medium High,±High1, as the scale of placement cases. The differencebetween two successive levels is equal to Low. The maxi-mum difference of the relative orderings of two alternativesis High. High (−High) represents the maximum satisfied(denied) level of satisfaction. In Section 4.1, we will explainthat the granularity of the scale is chosen according to thehumans’ cognitive abilities and constraints.

An Illustrative Example. In Figure 6, Usability (g1),Performance (g2), and Maintainability (g3) are the criteriafor deciding over two alternatives, the ONE-ITS (A1) andthe centralized data provider (A2). The goal graph at theright-hand side of Figure 6 shows the contributions of thesealternatives to the goals. Both alternatives have a nega-tive impact on Performance, but A1 has a Medium Highstronger impact on this goal than A2. This comparisondoes not indicate whether A1’s Performance is −High or−Low, but we know that whatever negative value that A1

contributes to Performance, A2 contributes Medium Highlevels less than that. Therefore, on the scale of −High toHigh, there are only two possible different sets of values thatcan be assigned to A1 and A2’s Performance. Either A1’sPerformance is −High (and A2’s Performance is −Low), orA1’s Performance is −Medium High (and A2’s Performanceis 0).

A1 has a negative impact on Maintainability and A2 hasa positive impact on Maintainability, while the difference ofthese two contributions is Low. Therefore, there are twopossible satisfaction levels for Maintainability of these twoalternatives: A1’s Maintainability is either −Low or 0 andaccordingly, A2’ Maintainability is 0 or Low. Since Usabilityof A1 and A2 has a High difference, there is only possibleUsability level for the alternatives: A1’s Usability iz 0 andA2’s Usability is High.

By combining 2 possible satisfaction levels for Performanceand Maintainability and the only possible level of Usability,

1In formulas, sets, and figurs, we will use the abbreviations0, L, ML, M, MH, and H respectively

655

Figure 6: Trade-offs of the ONE-ITS project on three simple goals and possible placement cases for the pairof alternatives

the algorithm generates four different goals satisfaction sta-tuses that one of them is the actual consequence of A1 andA2. The right-hand side of Figure 6 shows these four possi-ble placement cases, P1, P2, P3, P4. Each of these placementcases is a unique combination of contribution values on allgoals with respect to the relative ordering of A1 and A2 onthe interval scale. Each case includes one possible way toplace the contributions of A1 and A2 on the scale, takingthe relative ordering of the alternatives into account.

Step 3: Determining the Dominant Alternative foreach Placement Case

Once all possible placement cases are enumerated, the al-gorithm needs to determine the dominant alternative foreach placement. In this part, first we discuss why a utilityfunction is not useful for determining the dominant alterna-tive, and then, we explain how the Even Swaps multi-criteriadecision analysis method [1] finds the better solution of eachplacement.

A simple additive utility function can sum up goals’ satis-faction levels for a placement case. For example, the utilityof A1 in the first case in Figure 6 would be 0+0+−L = −L.In this way, the“value”of a goal satisfaction level is assumedto have a linear relation with the strength of contributionsto the goal. However, we discussed in Section 3 that valuefunctions are not necessarily a linear function of contribu-tions and strength of contributions does not reflect the sat-isfaction values and priorities of goals in the utility function.Hence, calculating a utility value is not a practical way fordetermining the dominant solution for a placement case.

To solve this problem, in [30], the Even Swaps methodis adapted for deciding on the optimum alternative designsolutions given the consequences of each alternative on a setof requirements. In an even swap, the decision analyst, col-laborating with the stakeholders, changes the contributionsof an alternative on one goal, and compensates this changewith a preferentially equal change in the satisfaction level ofanother goal. In other word, the Even Swaps makes trade-offs by asking decision stakeholders to give up on one NFRfor better satisfaction level of another NFR. For instance, inthe placement case 1 in Figure 6, where consequences of A1

on Uability, Performance, Maintainability is {0, 0,−L} andconsequences of A2 = {H,−MH, 0}, stakeholders agree thatif Performance of A1 is reduced from 0 to -MH, then Usabil-ity of A1 shall be increased from 0 to MH. This creates a

new virtual alternative like A′1 with revised consequences, sothe consequences of A′1 would be {MH,−MH,−L}. Notethat this swap shows how much stakeholders are willing tosacrify on Performance for better Usability, and the alterna-tives are not actually improved.

The virtual alternative is as preferred as the initial one,and it can be used as a surrogate. Then, the irrelevantgoals from the decision process, i.e., goals that are not dif-ferent between alternatives, are removed from the decisionanalysis process. For example A2 and A′1 are indifferent forPerformnace (g2,) so g2 can be removed from the problem.The underlying purpose of the swaps is to either make at-tributes irrelevant, in the sense that both alternatives haveequal consequences on this goal, or create a dominated al-ternative, in the sense that the other alternative is at leastas good as this alternative on every attribute. Such goalscan be eliminated, and the process continues until the mostpreferred alternative remains.

In the example we discussed earlier, decreasing g2 from0 to −MH is compensated with increasing g1 from 0 toMH; thus, consequences of A′1 = {MH,−MH,−L} andconsequences of A2 = {H,−MH, 0}. A2 has a higher (orequal) satisfaction level on all goals, therefore for the firstplacement case (P1), A2 is the dominant solution.

To analyze the second possible placement, P2, the analystasks stakeholders what is the value of x, if g3 or A1 is in-creased from 0 to L, and as the compensation, g2 is reducedfrom 0 to x. Stakeholders agree with x = −M , and accord-ingly, consequences of A′1 = {0,−M, L} and consequences ofA2 = {H,−MH, L}. So g3 is an irrelevant goal and can beremoved from the decision analysis over A1 and A2 for P2.Sat(A′1, g1) < Sat(A2, g1) but Sat(A′1, g1) > Sat(A2, g1),thus another swap is needed to decide which alternative isdominant. The analyst asks stakeholders what is the valueof x, if g2 is decreased from −M to −MH, and as the com-pensation, g1 is increased from 0 to x. Stakeholders agreewith x = −L; thus consequences of A′1 = {−L,−MH, L}and consequences of A2 = {H,−MH, L}, which indicatesthat A2 is the dominant solution for P2. Using the samemethod, A2 is recognized as the dominant solution for therest of the placement cases.

Step 4: Final Decision with the Human JudgmentInvolvement

By applying the Even Swaps method, an alternative might

656

be determined as the dominant solution for all placements.In that case, the alternative is proposed as the better solu-tion in the pair. For example, in Figure 6, A2 is determinedas the dominant solution in all of the placement cases, so A2

is the overall optimum solution. The analyst is still unawarewhich of those placement cases was the actual consequenceof the alternatives; nevertheless, the algorithm determinesthe best solution without the need to exact numerical satis-faction level of goals or any other extra information.

When an alternative is not dominant for all cases, but itis determined as the dominant one for the majority of theplacement cases, the algorithm finds the exceptional pat-terns of placements where the alternative is dominated bythe other solution. For example, consider two hypotheticalsolutions: A and B. An exceptional pattern may indicatean alternative like A is dominant unless usability of A islower than Medium. The experts’ judgment is then neededto examine the exceptional patterns, and either reject themas rare cases, or confirm that contributions’ strengths matchthe exceptional placement cases. The final decision is thenbased on this judgment. To decide between A and B, ifthe domain expert believes that the usability level of A isat least Medium or higher (not matched with the pattern),then A is definitely the overall better solution in the pair ofA and B. On the other hand, if the domain expert believesthat the usability of A is lower than Medium (matched withthe pattern), B is the optimum solution in the pair.

4.1 DiscussionMethod Benefits. By direct measurement of require-

ments satisfaction, m × n knowledge-intensive and cogni-tively hard queries are asked from the domain experts. Byapplying the proposed method, decision stakeholders needto answer m× (n− 1) comparison queries which require lesscognitive abilities. Although the suggested heuristic algo-rithm may not always provide a definitive answer, it enablesobjective trade-off decision making even though detailed nu-merical data is not available. The heuristic algorithm mayultimately rely on the human judgment, for which expertsare asked to judge whether the contribution values match theexceptional patterns. Nevertheless, such judgments requireless effort than providing absolute measures for satisfactionlevel of all goals. Extracting utility of alternatives and nu-merical importance weights and calculating a utility valueis less computionally complex and does not rely on humanjudgement for the final decision; however, even if one pieceof information in the utility formula is missing, the utilityvalue does not provide any answer.

Our Choice of the Scale. Detailed reliable input dataresult in reliable decisions. On the other hand, the moredetailed estimates we expect from stakeholders and experts,the more we may receive arbitrary numbers. We need tobalance the amount of information we request from stake-holders. Although it is desirable to acquire as much detailas possible, it is crucial to consider time and budget lim-itations for requirements analysis. At the same time, thecognitive abilities and knowledge required by the stakehold-ers and experts to provide the decision analysis input shouldnot be overestimated.

Saaty [31] argues that humans are able to distinguish be-tween high, medium, and low at two levels of comparison,which gives us nine categories from (low,low) to (high,high).The choice of the scale in the current work is based on this

humans’ cognitive ability (and constraint). The granularityof the scale affects the number of possible placement casesthat the algorithm examines, and consequently the numberof exceptional patterns for which stakeholders’ judgment isneeded. By using the scale of (low,low) to (high,high), inthe worst case, contributions of two alternatives may havea (low,low) difference on all goals. Hence, for each goal, thealternatives can be placed in 9 different places, which resultsin 9m placement cases (for m goals). The scale of 1-9 quicklybecomes inefficient, therefore, we use the scale of 0 to High(5 intervals) to limit the number of placement cases to theorder of 5m. Note that although the scale of −High to Highhas 11 different intervals, when the difference between twoalternatives is Low, at most 5 different placements (negativeor positive side of the scale) are possible.

Complexity Analysis. The heuristic trade-off analysismethod is implemented in a prototype tool that interactswith the user to get the comparisons, asks swaps, finds ex-ceptional patterns, and gets the human judegment about thepatterns. The running time of the algorithm is the functionof T (n, m, s), where n is the number of competing alterna-tives, m is the number of goals (criteria of comparisons), ands is the granularity of the comparisons (s is set to be 5 in thiswork as the highest difference between two contributions isHigh). In each cycle, the algorithm determines the domi-nant solution in a pair of alternatives; hence for n alterna-tives, n−1 cycles are needed: T (n, m, s) = (n−1)×Tp(m, 5).

Tp(m, 5) is the running time to generate all possible place-ment cases and decide on the dominant alternative. In theworst case, where the difference of a pair of alternativesis Low, the algorith generates 5 placement possibilities foreach goal, and given m goals, Tp(m, s) = 5 × 5... × 5 (mtimes)= O(5m). Although the algorithm complexity is inthe order of O(5m), in reality, m is limited to the numberof leaf goals in the goal graph which are requirements of asub-system that need to be considered for decision making.

Method Limitations. The major limitation of thismethod is the possibility of facing an alternative in the pairthat is dominant for half of the placement cases; thus stake-holders are required to examine numerous exceptional cases,which are the other half of the placement cases. Examin-ing those many exceptional cases is practically infeasible forhuman users. Another threat to validity of the method isusing the interval scale of −High to High for comparing thealternatives and expecting that stakeholders are able to dif-ferentiate the strength of contributions in this interval scale.The way that stakeholders understand the intervals of com-parisons may differ from the way in fact we use them in thealgorithm. The number of placement cases grow in the orderof 5m for m goals. When the problem scales, the algorithmmay return several exceptional patterns of satisfaction levelto the human expert for the final judgment.

Applying the Even Swaps to a large number of placementcases is tedious and labor-intensive. In practice, the stake-holders may not be able to select the right goals to swap.Making trade-offs by even swaps may also need relativelyextensive cognitive abilities and domain knowledge. Never-theless, if stakeholders are not able the specify the maximalamount of a goal satisfaction level that they are willing tosacrifice for a unit of increase in another goal, then it isprobable that they will not be able to numerically specifypreferences over the goals either.

657

5. THE ONE-ITS CASE STUDYWe applied the proposed method to the ONE-ITS project

that has been collaboratively developed by the Departmentof Civil Engineering at the University of Toronto, the Uni-versity of Regina, the Ministry of Transportation, Ontario(MTO), and the City of Toronto. In this case study, thegoal models were developed based on reviewing the projectdocuments and interviewing an MTO expert. In Figure 7,a goal model which focuses on the Management of VMS isgiven. In a separate interview session that took one hour, theMTO expert compared the alternative systems on a numberof goals. Figure 7 shows how the alternative solutions affectthe goals of the VMS application, and lists comparisons thatthe MTO expert provided.

The heuristic method generates 540 different placementcases based on the comparisons that MTO expert provided.The algorithm did not generate any exceptional patterns,and the ONE-ITS web-based system was determined as thebetter solution for all 540 possible placement cases. TheMTO expert stated that ONE-ITS is the better solutionin their opinion as well, which provides a support for thecorrectness of the decision suggested by our method.

Figure 7: Comparison of alternative solutions forthe VMS sub-system

6. CONCLUSIONS, LIMITATIONS, AND FU-TURE WORK

Qualitative goal model evaluation techniques are not ableto sufficiently differentiate alternative solutions. Quanti-tative MCDA methods for requirements trade-off decisionmaking rely on availability and accuracy of numerical esti-mates or measures of goal satisfaction levels. To avoid theseproblems and limitations, we propose a heuristic decisionmaking algorithm that collects relative orderings of alterna-tives to analyze requirements trade-off. Although the sug-gested heuristic method may not always find the optimumsolution or may rely on human judgment, it enables an algo-rithmic trade-off decision making even though detailed nu-merical data are not available.

We are enhacing the algorithm to automatically suggesteven swaps and reuse them in other placement cases. In anongoing work, we are exploring ways to reduce the number

of possible placement cases by adding thresholds and nu-merical measures of contributions to the model, wheneversuch data is available. This may reduce the number of ex-ceptional cases (patterns) that require a final judgment bystakeholders and experts. Further case studies and appli-cation of the method in action research and experiments isneeded for evaluating the utility and usability of the methodand the tool.

7. REFERENCES[1] J. S. Hammond, R. L. Keeney, and H. Raiffa, Smart choices : a practical guide

to making better life decisions. Broadway Books, 2002.[2] A. van Lamsweerde and E. Letier, “Handling obstacles in goal-oriented

requirements engineering,” IEEE Trans. Softw. Eng., vol. 26, no. 10, pp.978–1005, 2000.

[3] M. S. Feather, S. L. Cornford, K. A. Hicks, J. D. Kiper, and T. Menzies,“A broad, quantitative model for making early requirements decisions,”IEEE Software, vol. 25, pp. 49–56, 2008.

[4] H. P. In, D. Olson, and T. Rodgers, “Multi-criteria preference analysis forsystematic requirements negotiation,” in COMPSAC ’02, 2002, pp. 887–892.

[5] W. Ma, L. Liu, H. Xie, H. Zhang, and J. Yin, “Preference model drivenservices selection,” in Proc. of CAiSE’09, 2009, pp. 216–230.

[6] J. Karlsson and K. Ryan, “A cost-value approach for prioritizingrequirements,” IEEE Softw., vol. 14, no. 5, pp. 67–74, 1997.

[7] E. Letier and A. van Lamsweerde, “Reasoning about partial goalsatisfaction for requirements and design engineering,” in SIGSOFT’04/FSE-12, 2004, pp. 53–62.

[8] D. Gardner, The Science of Fear: Why We Fear the Things We Shouldn’t–and PutOurselves in Greater Danger. Dutton Adult, 2008.

[9] B. Schneier, “The psychology of security,” Commun. ACM, vol. 50, no. 5, p.128, 2007.

[10] L. Chung, B. A. Nixon, E. Yu, and J. Mylopoulos, Non-FunctionalRequirements in Software Engineering. Kluwer Academic, 1999.

[11] E. Yu, “Modeling Strategic Relationships for Process Reengineering,”Ph.D. dissertation, University of Toronto, 1995.

[12] T. L. Saaty, The Analytic Hierarchy Process: Planning, Priority Setting, ResourceAllocation. Mcgraw-Hill, 1980.

[13] V. Belton and T. J. Stewart, Multiple Criteria Decision Analysis: An IntegratedApproach. Springer, 2001.

[14] R. L. Keeney and H. Raiffa, Decisions with multiple objectives : preferences andvalue tradeoffs. Wiley, 1976.

[15] J. Figueira, S. Greco, and M. Ehrgott, Multiple Criteria Decision Analysis:State of the Art Surveys. Springer Verlag, 2005.

[16] J. Karlsson and K. Ryan, “A cost-value approach for prioritizingrequirements,” IEEE Softw., vol. 14, no. 5, pp. 67–74, 1997.

[17] I. S. Brito, F. Vieira, A. Moreira, and R. A. Ribeiro, “Handling conflicts inaspectual requirements compositions,” T. Aspect-Oriented SoftwareDevelopment, vol. 3, pp. 144–166, 2007.

[18] J. Yen and W. A. Tiao, “A systematic tradeoff analysis for conflictingimprecise requirements,” in RE ’97, 1997, p. 87.

[19] I. F. Alexander, “Initial industrial experience of misuse cases in trade-offanalysis,” in In Proc. of RE’02. IEEE Computer Society, 2002, pp. 61–70.

[20] L. Bass, P. Clements, and R. Kazman, Software Architecture in Practice.Second Edition, Addison Wesley, 2003.

[21] S. H. Houmb, J. Jurjens, G. Georg, and R. France, “An integrated securityverification and security solution trade-off analysis,” In Integrating Securityand Software Engineering, 2006.

[22] H. Kaiya, H. Horai, and M. Saeki, “AGORA: Attributed goal-orientedrequirements analysis method,” In RE’02, vol. 0, p. 13, 2002.

[23] J. Horkoff and E. Yu, “A Qualitative, Interactive Evaluation Procedure forGoal- and Agent-Oriented Models,” in CAiSE Forum. CEUR WorkshopProceedings, 2009.

[24] P. Giorgini, G. Manson, and H. Mouratidis, “On security requirementsanalysis for multi-agent systems.” in SELMAS’03, 2003.

[25] Y. Asnar and P. Giorgini, “Modelling risk and identifying countermeasurein organizations,” 2006, pp. 55–66.

[26] D. Amyot, S. Ghanavati, J. Horkoff, G. Mussbacher, L. Peyton, and E. Yu,“Evaluating goal models within the goal-oriented requirement language,”Int. Journal of Intelligent Systems (IJIS) (to appear), 2010.

[27] E. Letier and A. van Lamsweerde, “Reasoning about partial goalsatisfaction for requirements and design engineering,” SIGSOFT Softw. Eng.Notes, vol. 29, no. 6, pp. 53–62, 2004.

[28] P. Giorgini, J. Mylopoulos, and R. Sebastiani, “Goal-oriented requirementsanalysis and reasoning in the tropos methodology,” Eng. Appl. Artif. Intell.,vol. 18, no. 2, pp. 159–171, 2005.

[29] F. Lootsma, Multi-criteria decision analysis via ratio and difference judgement.Dordrecht: Kluwer Academic, 1999.

[30] G. Elahi and E. Yu, “Trust trade-off analysis for security requirementsengineering,” In Proc. of RE’09, vol. 0, pp. 243–248, 2009.

[31] T. Saaty, “The analytic hierarchy and analytic network processes for themeasurement of intangible criteria and for decision-making,” in MultipleCriteria Decision Analysis: State of the Art Surveys. Springer Verlag, 2005, pp.345–408.

658