decision making (1)
DESCRIPTION
TRANSCRIPT
Decision AnalysisDecision Analysisand Tradeoff Studiesand Tradeoff Studies
Terry BahillTerry BahillSystems and Industrial EngineeringSystems and Industrial EngineeringUniversity of ArizonaUniversity of [email protected]@sie.arizona.edu©, 2000-10, Bahill©, 2000-10, BahillThis file is located in This file is located in http://www.sie.arizona.edu/sysengr/slides/http://www.sie.arizona.edu/sysengr/slides/
04/10/23 © 2009 Bahill
2
AcknowledgementAcknowledgementThis research was supported by AFOSR/MURI F49620-03-1-0377.
04/10/23 © 2009 Bahill
3
Timing estimate for this course*Timing estimate for this course*• Introduction (10 minutes)
• Decision analysis and resolution (49 slides, 40 minutes)
• San Diego Airport example (7 slides, 5 minutes)
• The tradeoff study process and potential problems (238 slides, 145 minutes)
• Summary (6 slides, 10 minutes)
• Dog system exercise (140 minutes)
• Mathematical summary of tradeoff methods (38 slides, 70 minutes)
• Course summary (10 minutes)
• Breaks (50 minutes)
• Total (480 minutes)
04/10/23 © 2009 Bahill
4
OutlineOutline**
•This course starts with brief model of human decision making (slides 14-27). Then it presents a crisp description of the tradeoff study processes (Slides 14-67), which includes a simple example of choosing between two combining methods.
•Then it shows a complex, but well-known tradeoff study example that most people will be familiar with: the San Diego airport site selection (Slides 68-75).
•Then we go back and examine many difficulties that could arise when designing a tradeoff study; we show many methods that have been used to overcome these potential problems (Slides 76-338).
•The course is summarized with slides 339-346.• In the Dog System Exercise, students create their own solutions for
a tradeoff study. These exercises will be computer based. The students complete one of the exercise’s eight parts. Then we give them our solutions. They complete another portion and we give them another solution. The computers will be preloaded with all of the problems and solutions. The students will use Excel spreadsheets and a simple program for graphing scoring (utility) functions.
•After the exercise there will be a mathematical summary of tradeoff methods. Students who are algebraically challenged may excuse themselves.
04/10/23 © 2009 Bahill
5
Course administrationCourse administration•AWO:
•Course Name: Decision Making
and Tradeoff Studies
•Course Number:
•FacilitiesTelephones*BathroomsVending MachinesExits
ExitExit
04/10/23 © 2009 Bahill
6
Course objectivesCourse objectives**
•The students should be able to Understand human decision making Use many techniques, including tradeoff
studies, to help select among alternatives Decide whether a problem is a good
candidate for a tradeoff study Establish evaluation criteria with weights of
importance Understand scoring (utility) functions Perform a valid tradeoff study Fix the do nothing problem Use several different combining functions Perform a sensitivity analysis Be aware of many tradeoff methods Develop a decision tree
04/10/23 © 2009 Bahill
7
Student introductionsStudent introductions•Name
•Current program assignment
•Related experience
Decision Analysis Decision Analysis and Resolutionand Resolution
04/10/23 © 2009 Bahill
9
CMMICMMI•The Capability Maturity Model
Integrated (CMMI) is a collection of best practices from diverse engineering companies
• Improvements to our organization will come from process improvements, not from people improvements or technology improvements
• CMMI provides guidance for improving an organization’s processes
•One of the CMMI process areas is Decision Analysis and Resolution (DAR)
04/10/23 © 2009 Bahill
10
DARDAR•Programs and Departments select the
decision problems that require DAR and incorporate them in their plans (e.g. SEMPs)
•DAR is a common process•Common processes are tools that the user
gets, tailors and uses•DAR is invoked throughout the whole
program lifecycle whenever a critical decision is to be made
•DAR is invoked by IPT leads on programs, financial analysts, program core teams, etc.
• Invoke the DAR Process in work instructions, in gate reviews, in phase reviews or with other triggers, which can be used anytime in the system life cycle
04/10/23 © 2009 Bahill
11
Typical decisionsTypical decisions• Decision problems that may require a formal
decision process Tradeoff studies Bid/no-bid Make-reuse-buy Formal inspection versus checklist
inspection Tool and vendor selection Cost estimating Incipient architectural design Hiring and promotions Helping your customer to choose a
solution
04/10/23 © 2009 Bahill
12
It’s not done just onceIt’s not done just once•A tradeoff study is not something that you do once at the beginning of a project.
•Throughout a project you are continually making tradeoffs creating team communication methods selecting components choosing implementation techniques designing test programsmaintaining schedule
•Many of these tradeoffs should be formally documented.
04/10/23 © 2009 Bahill
13
PurposePurpose**
“In all decisions you gain something and lose something. Know what they are and do it deliberately.”
04/10/23 © 2009 Bahill
14
Tradeoff StudiesTradeoff Studies
04/10/23 © 2009 Bahill
15
A simple tradeoff studyA simple tradeoff study
04/10/23 © 2009 Bahill
16
DAR
Specific Practice
Decide if formal evaluation is needed
When to do a tradeoff study
Establish Evaluation Criteria
What is in a tradeoff study
Identify Alternative Solutions
Select Evaluation Methods
Evaluate Alternatives
Select Preferred Solutions
CMMI’s DAR processCMMI’s DAR process
04/10/23 © 2009 Bahill
17
Tradeoff Study ProcessTradeoff Study Process**
These tasks are drawn serially,but they are not performed in a serial manner. Rather, it is an iterative processwith many feedback loops, which are not shown.
Decide if FormalEvaluation is
Needed
Decide if FormalEvaluation is
Needed
Problem StatementProblem
Statement
SelectEvaluation Methods
SelectEvaluation Methods
Establish Evaluation
Criteria
Establish Evaluation
Criteria
Identify AlternativeSolutions
Identify AlternativeSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateAlternatives
EvaluateAlternatives
Select PreferredSolutions
Select PreferredSolutions
Formal Evaluations
Formal Evaluations
PerformExpert Review
PerformExpert Review
Preferred SolutionsPreferred Solutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
∑
04/10/23 © 2009 Bahill
18
When creating a processWhen creating a process
the most important facets are•illustrating tasks that can be done in
parallel•suggesting feedback loops•configuration management•including a process to improve the
process
04/10/23 © 2009 Bahill
19
Humans make four types of decisions:Humans make four types of decisions:•Allocating resources among competing projects* •Generating plans, schedules and novel ideas•Negotiating agreements•Choosing amongst alternatives Alternatives can be examined in series or
parallel. When examined in series it is called sequential
search When examined in parallel it is called a tradeoff
or a trade study “Tradeoff studies address a range of
problems from selecting high-level system architecture to selecting a specific piece of commercial off the shelf hardware or software. Tradeoff studies are typical outputs of formal evaluation processes.”*
04/10/23 © 2009 Bahill
20
HistoryHistoryBen Franklin’s letter* to Joseph Priestly outlined one of the first descriptions of a tradeoff study.
04/10/23 © 2009 Bahill
21
Decide if Formal Evaluation is NeededDecide if Formal Evaluation is Needed
Decide ifDecide if FormalFormalEvaluation isEvaluation is
Needed Needed
Problem StatementProblem
Statement
SelectEvaluation Methods
SelectEvaluation Methods
Establish Evaluation
Criteria
Establish Evaluation
Criteria
Identify AlternativeSolutions
Identify AlternativeSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateAlternatives
EvaluateAlternatives
Select PreferredSolutions
Select PreferredSolutions
Formal Evaluations
Formal Evaluations
PerformExpert Review
PerformExpert Review
Preferred SolutionsPreferred Solutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
04/10/23 © 2009 Bahill
22
Is formal evaluation needed?Is formal evaluation needed?Companies should have polices for when to do
formal decision analysis. Criteria include• When the decision is related to a moderate or high-
risk issue
• When the decision affects work products under configuration management
• When the result of the decision could cause significant schedule delays
• When the result of the decision could cause significant cost overruns
• On material procurement of the 20 percent of the parts that constitute 80 percent of the total material costs
04/10/23 © 2009 Bahill
23
Guidelines for formal evaluationGuidelines for formal evaluation• When the decision is selecting one or a few
alternatives from a list• When a decision is related to major changes in
work products that have been baselined• When a decision affects the ability to achieve
project objectives• When the cost of the formal evaluation is
reasonable when compared to the decision’s impact
• On design-implementation decisions when technical performance failure may cause a catastrophic failure
• On decisions with the potential to significantly reduce design risk, engineering changes, cycle time or production costs
04/10/23 © 2009 Bahill
24
Establish Evaluation CriteriaEstablish Evaluation Criteria
Decide if FormalEvaluation is
Needed
Decide if FormalEvaluation is
Needed
Problem StatementProblem
Statement
SelectEvaluation Methods
SelectEvaluation Methods
Establish Establish EvaluationEvaluation
CriteriaCriteria
Identify AlternativeSolutions
Identify AlternativeSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateAlternatives
EvaluateAlternatives
Select PreferredSolutions
Select PreferredSolutions
Formal Evaluations
Formal Evaluations
PerformExpert Review
PerformExpert Review
Preferred SolutionsPreferred Solutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
04/10/23 © 2009 Bahill
25
Establish evaluation criteriaEstablish evaluation criteria**
•Establish and maintain criteria for evaluating alternatives
•Each criterion must have a weight of importance•Each criterion should link to a tradeoff
requirement, i.e. a requirement whose acceptable value can be more or less depending on quantitative values of other requirements.
•Criteria must be arranged hierarchically. The top-level may be performance, cost, schedule and risk. Program Management should prioritize these
four criteria at the beginning of the project and make sure everyone knows the priorities.
•All companies should have a repository of generic evaluation criteria.
04/10/23 © 2009 Bahill
26
What will you eat for lunch today?What will you eat for lunch today?•In class exercise.
•Write some evaluation criteria that will, help you decide.*
04/10/23 © 2009 Bahill
27
Killer tradesKiller trades•Evaluating alternatives is expensive.
•Therefore, early in tradeoff study, identify very important requirements* that can eliminate many alternatives.
•These requirements produce killer criteria.**
•Subsequent killer trades can often eliminate 90% of the possible alternatives.
04/10/23 © 2009 Bahill
28
Identify Alternative SolutionsIdentify Alternative Solutions
Decide if FormalEvaluation is
Needed
Decide if FormalEvaluation is
Needed
Problem StatementProblem
Statement
SelectEvaluation Methods
SelectEvaluation Methods
Establish Evaluation
Criteria
Establish Evaluation
Criteria
Identify Identify AlternativeAlternativeSolutionsSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateAlternatives
EvaluateAlternatives
Select PreferredSolutions
Select PreferredSolutions
Formal Evaluations
Formal Evaluations
PerformExpert Review
PerformExpert Review
Preferred SolutionsPreferred Solutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
04/10/23 © 2009 Bahill
29
Identify alternative solutionsIdentify alternative solutions• Identify alternative solutions for the
problem statement
• Consider unusual alternatives in order to test the system requirements*
• Do not list alternatives that do not satisfy all mandatory requirements**
• Consider use of commercial off the shelf and in-house entities***
• Use killer trades to eliminate thousands of infeasible alternatives
04/10/23 © 2009 Bahill
30
What will you eat for lunch today?What will you eat for lunch today?•In class exercise.
•List some alternatives for today’s lunch.*
04/10/23 © 2009 Bahill
31
Select Evaluation MethodsSelect Evaluation Methods
Decide if FormalEvaluation is
Needed
Decide if FormalEvaluation is
Needed
Problem StatementProblem
Statement
SelectSelectEvaluation Evaluation MethodsMethods
Establish Evaluation
Criteria
Establish Evaluation
Criteria
Identify AlternativeSolutions
Identify AlternativeSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateAlternatives
EvaluateAlternatives
Select PreferredSolutions
Select PreferredSolutions
Formal Evaluations
Formal Evaluations
PerformExpert Review
PerformExpert Review
Preferred SolutionsPreferred Solutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
04/10/23 © 2009 Bahill
32
Select evaluation methodsSelect evaluation methods• Select the source of the evaluation data and the
method for evaluating the data• Typical sources for evaluation data include
approximations, product literature, analysis, models, simulations, experiments and prototypes*
• Methods for combining data and evaluating alternatives include Multi-Attribute Utility Technique (MAUT), Ideal Point, Search Beam, Fuzzy Databases, Decision Trees, Expected Utility, Pair-wise Comparisons, Analytic Hierarchy Process (AHP), Financial Analysis, Simulation, Monte Carlo, Linear Programming, Design of Experiments, Group Techniques, Quality Function Deployment (QFD), radar charts, forming a consensus and Tradeoff Studies
04/10/23 © 2009 Bahill
33
Collect evaluation dataCollect evaluation data•Using the appropriate source (approximations, product literature, analysis, models, simulations, experiments or prototypes) collect data for evaluating each alternative.
04/10/23 © 2009 Bahill
34
Evaluate AlternativesEvaluate Alternatives
Decide if FormalEvaluation is
Needed
Decide if FormalEvaluation is
Needed
Problem StatementProblem
Statement
SelectEvaluation Methods
SelectEvaluation Methods
Establish Evaluation
Criteria
Establish Evaluation
Criteria
Identify AlternativeSolutions
Identify AlternativeSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateEvaluateAlternativesAlternatives
Select PreferredSolutions
Select PreferredSolutions
Formal Evaluations
Formal Evaluations
PerformExpert Review
PerformExpert Review
Preferred SolutionsPreferred Solutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
04/10/23 © 2009 Bahill
35
Evaluate alternativesEvaluate alternatives•Evaluate alternative solutions using the evaluation criteria, weights of importance, evaluation data, scoring functions and combining functions.
•Evaluating alternative solutions involves analysis, discussion and review. Iterative cycles of analysis are sometimes necessary. Supporting analyses, experimentation, prototyping, or simulations may be needed to substantiate scoring and conclusions.
04/10/23 © 2009 Bahill
36
Select Preferred SolutionsSelect Preferred Solutions
Decide if FormalEvaluation is
Needed
Decide if FormalEvaluation is
Needed
Problem StatementProblem
Statement
SelectEvaluation Methods
SelectEvaluation Methods
Establish Evaluation
Criteria
Establish Evaluation
Criteria
Identify AlternativeSolutions
Identify AlternativeSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateAlternatives
EvaluateAlternatives
Select Select PreferredPreferredSolutionsSolutions
Formal Evaluations
Formal Evaluations
PerformExpert Review
PerformExpert Review
Preferred Preferred SolutionsSolutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
04/10/23 © 2009 Bahill
37
Select preferred solutionsSelect preferred solutions• Select preferred solutions from the alternatives
based on evaluation criteria.
• Selecting preferred alternatives involves weighing and combining the results from the evaluation of alternatives. Many combining methods are available.
• The true value of a formal decision process might not be listing the preferred alternatives. More important outputs are stimulating thought processes and documenting their outcomes.
• A sensitivity analysis will help validate your recommendations.
• The least sensitive criteria should be given weights of 0.
04/10/23 © 2009 Bahill
38
Perform Expert ReviewPerform Expert Review
Decide if FormalEvaluation is
Needed
Decide if FormalEvaluation is
Needed
Problem StatementProblem
Statement
SelectEvaluation Methods
SelectEvaluation Methods
Establish Evaluation
Criteria
Establish Evaluation
Criteria
Identify AlternativeSolutions
Identify AlternativeSolutions
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
EvaluateAlternatives
EvaluateAlternatives
Select PreferredSolutions
Select PreferredSolutions
Formal Evaluations
Formal Evaluations
Perform Expert Review
Perform Expert Review
Preferred SolutionsPreferred Solutions
Present ResultsPresent Results
Put In PPAL
Put In PPAL
∑
04/10/23 © 2009 Bahill
39
Perform expert reviewPerform expert review11
• Formal evaluations should be reviewed* at regular gate reviews such as SRR, PDR and CDR or by special expert reviews
• Technical reviews started about the same time as Systems Engineering, in 1960. The concept was formalized with MIL-STD-1521 in 1972.
• Technical reviews are still around, because there is evidence that they help produce better systems at less cost.
04/10/23 © 2009 Bahill
40
Perform expert reviewPerform expert review22
•Technical reviews evaluate the product of an IPT*
•They are conducted by a knowledgeable board of specialists including supplier and customer representatives
•The number of board members should be less than the number of IPT members
•But board expertise should be greater than the IPT’s experience base
04/10/23 © 2009 Bahill
41
Who should come to the review?Who should come to the review?•Program Manager•Chief Systems Engineer•Review Inspector•Lead Systems Engineer•Domain Experts• IPT Lead•Facilitator •Stakeholders for this decision
Builder Customer Designer Tester PC Server
•Depending on the decision, the Lead Hardware Engineer and the Lead Software Engineer
04/10/23 © 2009 Bahill
42
Present resultsPresent resultsPresent the results* of the formal evaluation to the original decision maker and other relevant stakeholders.
04/10/23 © 2009 Bahill
43
Put in the PALPut in the PAL• Formal evaluations reviewed by experts
should be put in the organizational Process Asset Library (PAL) or the Project Process Asset Library (PPAL)
• Evaluation data for tradeoff studies come from approximations, analysis, models, simulations, experiments and prototypes. Each time better data is obtained the PAL should be updated.
• Formal evaluations should be designed with reuse in mind.
04/10/23 © 2009 Bahill
44
Closed Book Quiz, 5 minutes Closed Book Quiz, 5 minutes Fill in the empty boxesFill in the empty boxes
Problem StatementProblem
Statement
ProposedAlternativesProposed
Alternatives
EvaluationCriteria
EvaluationCriteria
Formal Evaluations
Formal Evaluations
Preferred SolutionsPreferred Solutions∑
04/10/23 © 2009 Bahill
45
Tradeoff Study ExampleTradeoff Study Example
04/10/23 © 2009 Bahill
46
Example: What method should Example: What method should we use for evaluating alternatives?we use for evaluating alternatives?**
• Is formal evaluation needed? Check the Guidance for Formal Evaluations We find that many of its criteria are satisfied
including “On decisions with the potential to significantly reduce design risk … cycle time ...”
Establish evaluation criteria Ease of Use Familiarity
Killer criterion Engineers must think that use of the technique
is intuitive.
04/10/23 © 2009 Bahill
47
Example (continued)Example (continued)11
• Identify alternative solutions Linear addition of weight times scores,
Multiattribute Utility Theory (MAUT).* This method is often called a “trade study.” It is often implemented with an Excel spreadsheet. Analytic Hierarchy Process (AHP)**
04/10/23 © 2009 Bahill
48
Example (continued)Example (continued)22
• Select evaluation methods The evaluation data will come from expert
opinion Common methods for combining data and
evaluating alternatives include: Multi-Attribute Utility Technique (MAUT),
Decision Trees, Analytic Hierarchy Process (AHP), Pair-wise Comparisons, Ideal Point, Search Beam, etc.
In the following slides we will use two methods: linear addition of weight times scores (MAUT) and the Analytic Hierarchy Process (AHP)*
04/10/23 © 2009 Bahill
49
Example (continued)Example (continued)33
• Evaluate alternatives Let the weights and evaluation data be
integers between 1 and 10, with 10 being the best. The computer can normalize the weights if necessary.
04/10/23 © 2009 Bahill
50
Multi-Attribute Utility Technique (MAUT)Multi-Attribute Utility Technique (MAUT)11
Criteria Weight of
Importance MAUT AHP
Ease of Use 8 4 Familiarity Sum of weight times score
Assess evaluation data* row by row
04/10/23 © 2009 Bahill
51
Multi-Attribute Utility Technique (MAUT)Multi-Attribute Utility Technique (MAUT)22
Criteria Weight* of Importance
MAUT AHP
Ease of Use 9 8 4 Familiarity 3 9 2 Sum of weight times score
99 42
The
winner
04/10/23 © 2009 Bahill
52
Analytic Hierarchy Process (AHP)Analytic Hierarchy Process (AHP)
Verbal scale Numerical
value Equally important, likely or preferred
1
Moderately more important, likely or preferred
3
Strongly more important, likely or preferred
5
Very strongly more important, likely or preferred
7
Extremely more important, likely or preferred
9
Verbal scale Numerical
value Equally important, likely or preferred
1
Moderately more important, likely or preferred
3
Strongly more important, likely or preferred
5
Very strongly more important, likely or preferred
7
Extremely more important, likely or preferred
9
04/10/23 © 2009 Bahill
53
AHP, make comparisonsAHP, make comparisonsCreate a matrix with the criteria on the diagonal and make pair-wise comparisons*Ease of Use Ease of Use is
moderately more important than Familiarity (3)
Reciprocal of 3 = 1/3 Familiarity
04/10/23 © 2009 Bahill
54
AHP, compute weightsAHP, compute weights• Create a matrix
• Square the matrix
• Add the rows
• Normalize*
1 1 23 3 3
1 3 1 3 2 6 8
1 1 2 2
0.7
. 5.6
5
0 27
1 1 23 3 3
1 3 1 3 2 6 8
1 1 2 2
0.7
. 5.6
5
0 27
04/10/23 © 2009 Bahill
55
In-class exerciseIn-class exercise•Use these criteria to help select your lunch today.Closeness, distance to the venue. Is it in the same building, the next building or do you have to get in a car and drive?Tastiness, including gustatory delightfulness, healthiness, novelty and savoriness.Price,* total purchase price including tax and tip.
04/10/23 © 2009 Bahill
56
To help select lunch todayTo help select lunch today11
•closeness is ??? more important than tastiness,
•closeness is ??? more important than price,
•tastiness is ??? more important than price.
Closeness Tastiness
Price
Closeness
Tastiness
Price
04/10/23 © 2009 Bahill
57
To help select lunch todayTo help select lunch today22
•closeness is strongly more important (5) than tastiness,
•closeness is very strongly more important (7) than price,
•tastiness is moderately more important (3) than price.
Closeness Tastiness
Price
Closeness 1 5 7
Tastiness 1 3
Price 1
04/10/23 © 2009 Bahill
58
To help select lunch todayTo help select lunch today33
1 5 7 1 5 7
3 12.3 29 44.3 0.731 1
1 3 1 3 0.8 3 7.4 11.2 0.195 5
0.4 1.4 3 4.8 0.081 1 1 1
1 17 3 7 3
Closeness Tastiness Price Weight of Importance
Closeness 1 5 7 0.73
Tastiness 1/5 1 3 0.19
Price 1/7 1/3 1 0.08
04/10/23 © 2009 Bahill
59
AHP, get scoresAHP, get scores Compare each alternative on the first criterion
1 12 2
1 2 1 2 2 4 6
1 1 1 2 3
0.67
0.33
1 12 2
1 2 1 2 2 4 6
1 1 1 2 3
0.67
0.33
Ease of Use MAUT In terms of Ease
of Use, MAUT is slightly preferred (2)
1/2 AHP
04/10/23 © 2009 Bahill
60
AHP, get scoresAHP, get scores22
Compare each alternative on the second criterion
1 15 5
1 5 1 5 2 10 0.83
0.17
12
1 1 0.4 2 2.4
1 15 5
1 5 1 5 2 10 0.83
0.17
12
1 1 0.4 2 2.4
Familiarity MAUT In terms of
Familiarity, MAUT is strongly preferred (5)
1/5 AHP
04/10/23 © 2009 Bahill
61
AHP, form comparison matrixAHP, form comparison matrix****
Combine with linear addition*
Criteria Weight of
Importance MAUT AHP
Ease of Use 0.75 0.67 0.33 Familiarity 0.25 0.83 0.17 Sum of weight times score
0.71 0.29
The
winner
04/10/23 © 2009 Bahill
62
Example (continued)Example (continued)44
•Select Preferred Solutions Linear addition of weight times scores
(MAUT) was the preferred alternative Now consider new criteria, such as
Repeatability of Result, Consistency*, Time to Compute Do a sensitivity analysis
04/10/23 © 2009 Bahill
63
Sensitivity analysis, simpleSensitivity analysis, simpleIn terms of Familiarity, MAUT was strongly preferred (5) over the AHP. Now change this 5 to a 3 and to a 7.
• Changing the scores for Familiarity does not change the recommended alternative.
• This is good.• It means the Tradeoff study is robust with
respect to these scores.
Final Score Familiarity MAUT AHP
3 0.69 0.31 5 0.71 0.29 7 0.72 0.28
04/10/23 © 2009 Bahill
64
Sensitivity analysis, analyticSensitivity analysis, analyticCompute the six semirelative-sensitivity functions, which are defined as
which reads, the semirelative-sensitivity function of the performance index F with respect to the parameter is the partial derivative of F with respect to times with everything evaluated at the normal operating point (NOP).
F
NOP
FS
FNOP
FS
04/10/23 © 2009 Bahill
65
Sensitivity analysisSensitivity analysis22
For the performance index use the alternative rating for MAUT minus the alternative rating for AHP*
F = F1 - F2 = Wt1×S11 + Wt2×S21 – Wt1×S12 –Wt2×S22
Criteria Weight of
Importance MAUT AHP
Ease of Use Wt1 S11 S12 Familiarity Wt2 S21 S22 Sum of weight times score
F1 F2
04/10/23 © 2009 Bahill
66
Sensitivity analysisSensitivity analysis33
The semirelative-sensitivity functions*
1
2
11
21
12
22
11 12 1
21 22 2
1 11
2 21
1 12
2 22
0.26
0.16
0.50
0.21
-0.25
-0.04
FWt
FWt
FS
FS
FS
FS
S S S Wt
S S S Wt
S Wt S
S Wt S
S Wt S
S Wt S
S11 is the most importantparameter. So go back and reevaluate it.
04/10/23 © 2009 Bahill
67
Sensitivity analysisSensitivity analysis44
•The most important parameter is the score for MAUT on the criterion Ease of Use
•We should go back and re-evaluate the derivation of that score
Ease of Use MAUT In terms of Ease
of Use, MAUT is slightly preferred (2)
1/2 AHP
04/10/23 © 2009 Bahill
68
The Decision Analysis and Resolution (DAR) Process
SelectEvaluationMethods
EvaluateAlternatives
PreferredSolutions
SelectSolutions
EstablishEvaluation
Criteria
EvaluationCriteria
IdentifyAlternativeSolutions
ProposedAlternatives
SelectionProblem
Decide if Formal
Evaluation Process is Warranted
ProblemStatement S
Manage the DAR process
Recommendations
FormalEvaluationsThese tasks are drawn
serially, but they are not performed in a serial manner. Rather it is an iterative process with many unshown feedback loops.
Decision to Not Proceed
ExpertReview
Put in PAL
Present Results to Decision
Maker
04/10/23 © 2009 Bahill
69
Example (continued)Example (continued)55
• Perform expert review of the tradeoff study.
• Present results to original decision maker.
• Put tradeoff study in PAL.
• Improve the DAR process. Add some other techniques, such as AHP, to
the DAR web course Fix the utility curves document Add image theory to the DAR process Change linkages in the documentation system Create a course, Decision Making and Tradeoff
Studies
04/10/23 © 2009 Bahill
70
Quintessential exampleQuintessential exampleA Tradeoff Study of Tradeoff Study Tools
is available at
http://www.sie.arizona.edu/sysengr/sie554/tradeoffStudyOfTradeoffStudyTools.doc
San Diego County San Diego County Regional Airport Regional Airport Tradeoff StudyTradeoff Study
This tradeoff study has cost $17 million.This tradeoff study has cost $17 million.
http://www.san.org/authority/assp/index.asp
http://www.san.org/airport_authority/archives/index.asp#master_plan
04/10/23 © 2009 Bahill
72
The evaluation criteria treeThe evaluation criteria tree**
Operational RequirementOptimal Airport LayoutRunway Alignment
TerrainWeatherExisting land uses
Wildlife HazardsJoint Use and National Defense CompatibilityExpandability
Ground AccessTravel Time, percentage of population in three travel time segments Roadway Network Capacity, existing and projected daily roadway volumes Highway and Transit Accessibility, distance to existing and planned
freeways Environmental Impacts
Quantity of residential land to be displaced by the airport developmentNoise Impact, population within each of three specific decibel ranges Biological Resources
Wetlands Protected speciesWater qualitySignificant cultural resources
Site Development Evaluations
04/10/23 © 2009 Bahill
73
Top-level criteriaTop-level criteria1.Operational Requirements
2.Ground Access
3.Environmental Impacts
4.Site Development Evaluations
These four evaluation criteria are then decomposed into a hierarchy
04/10/23 © 2009 Bahill
74
Operational RequirementsOperational RequirementsOptimal Airport LayoutRunway Alignment Terrain, weather and existing land uses
Wildlife Hazards Joint Use and National Defense CompatibilityExpandability
04/10/23 © 2009 Bahill
75
Ground AccessGround Access• Travel Time, percentage of population in
three travel time segments
• Roadway Network Capacity, existing and projected daily roadway volumes
• Highway and Transit Accessibility, distance to existing and planned freeways
04/10/23 © 2009 Bahill
76
Environmental ImpactsEnvironmental Impacts•Quantity of residential land to be displaced by the airport development
•Noise Impact, population within each of three specific decibel ranges
•Biological ResourcesWetlands Protected species
•Water quality
•Significant cultural resources
04/10/23 © 2009 Bahill
77
Alternative LocationsAlternative Locations•Miramar Marine Corps Air Station
•East Miramar
•North Island Naval Air Station
•March Air Force Base
•Marine Corps Base Camp Pendleton
• Imperial County desert site
•Campo and Borrego Springs
•Lindberg Field
•Off-Shore floating airport
•Corte Madera Valley
04/10/23 © 2009 Bahill
78
Tradeoff Studies: Tradeoff Studies: the Process and Potential the Process and Potential
ProblemsProblems**
04/10/23 © 2009 Bahill
80
Outline of this sectionOutline of this section• Problem statement• Models of human decision making• Components of a tradeoff study
Problem statement Evaluation criteria Weights of importance Alternative solutions
The do nothing alternative Different distributions of alternatives
Evaluation data Scoring functions Scores Combining functions Preferred alternatives Sensitivity analysis
• Other tradeoff techniques The ideal point The search beam Fuzzy sets Decision trees
• The wrong answer• Tradeoff study on tradeoff study tools• Summary
04/10/23 © 2009 Bahill
81
ReferenceReferenceJ. Daniels, P. W. Werner and A. T. Bahill, Quantitative Methods for Tradeoff Analyses, Systems Engineering, 4(3), 199-212, 2001.
04/10/23 © 2009 Bahill
82
PurposePurposeThe systems engineer’s job is to elucidate domain knowledge and capture the values and preferences of the decision maker, so that the decision maker (and other stakeholders) will have confidence in the decision.
The decision maker balances effort with confidence*
04/10/23 © 2009 Bahill
83
04/10/23 © 2009 Bahill
84
Tradeoff studiesTradeoff studies•Humans exhibit four types of decision making
activities
1. Allocating resources among competing projects
2. Making plans, which includes scheduling
3. Negotiating agreements
4. Choosing alternatives from a list Series
Parallel, a tradeoff study
04/10/23 © 2009 Bahill
85
A typical tradeoff study matrix Alternative-A Alternative-B Criteria Qualitative
weight Normalized weight
Scoring function
Input value
Output score
Score times weight
Input value
Output score
Score times weight
Criterion-1 1 to 10 0 to 1 Type and parameters
Natural units
0 to 1 0 to 1 Natural units
0 to 1 0 to 1
Criterion-2 1 to 10 0 to 1 Type and parameters
Natural units
0 to1 0 to 1 Natural units
0 to1 0 to 1
Sum 0 to1 0 to1
04/10/23 © 2009 Bahill
86
Pinewood Derby*
04/10/23 © 2009 Bahill
87
Part of a Pinewood Derby tradeoff studyPart of a Pinewood Derby tradeoff studyPerformance figures of merit evaluated on a prototype for a Round Robin with Best Time Scoring
Evaluation criteria
Input value
Score Weight Score times
weight 1. Average Races
per Car 6 0.94 0.20 0.19
2. Number of Ties 0 1 0.20 0.20 3. Happiness 0.87 0.60 0.52
Qualitative
weight Normalized
weight Input value
Scoring function
Output score
Score times
weight
3.1 Percent Happy Scouts
10 0.50 96
0.98
96
0.98 0.49
3.2 Number of Irate Parents
5 0.25 1
1
0.5
1
0.5
0.50 0.13
3.3 Number of Lane Repeats
5 0.25 0
1.0
0
1.00 0.25
Sum 0.87 0.91
http://www.sie.arizona.edu/sysengr/pinewood/pinewood.pdf
04/10/23 © 2009 Bahill
88
When do people do tradeoff studies?When do people do tradeoff studies?•Buying a car
•Buying a house
•Selecting a job
•These decisions are important, you have lots of time to make the decision and alternatives are apparent.*
•We would not use a tradeoff study to select a drink for lunch or to select a husband or wife.
•You would also do a tradeoff study when your boss asks you to do one.
04/10/23 © 2009 Bahill
89
Do the tradeoff studies upfront Do the tradeoff studies upfront before all of the costs are locked inbefore all of the costs are locked in**
100
80
20
0
60
Co
st (
%)
TimeConceptdevelopment
Full-scale
design
Start ofproduction
40
Actualexpenditures
Final costs locked-in
04/10/23 © 2009 Bahill
90
Why discuss this topic?Why discuss this topic?• Many multicriterion decision-making
techniques exist, but few decision-makers use them.
• Perhaps, because They seem complicated Different techniques have given different
preferred alternatives Different life experiences give different
preferred alternatives People don’t think that way*
04/10/23 © 2009 Bahill
91
Models of Human Decision MakingModels of Human Decision Making
04/10/23 © 2009 Bahill
92
Series versus parallelSeries versus parallel11
• Looking at alternatives in parallel is not an innate human action.
• Usually people select one hypothesis and work on it until it is disproved, then they switch to a new alternative: that’s the scientific method.
• Such serial processing of alternatives has been demonstrated for Fire fighters Airline pilots Physicians Detectives Baseball managers People looking for restaurants*
04/10/23 © 2009 Bahill
93
Series versus parallelSeries versus parallel22
•V. V. Krishnan has a model of animals searching for habitat (home, breeding area, hunting area, etc.)
•It uses the value of each habitat and the cost of moving between sites.
•When travel between sites is inexpensive, e. g. birds or honeybees* searching for a nest site, the search is often a tradeoff study comparing alternatives in parallel.
•When travel is expensive, e.g. beavers searching for a dam site, the search is usually sequential.
04/10/23 © 2009 Bahill
94
Series versus parallelSeries versus parallel33**
•If a person is looking for a new car, he or she might perform a tradeoff study.
•Whereas a person looking for a used car might use a sequential search, because the availability of cars would change day by day.
04/10/23 © 2009 Bahill
95
The need for changeThe need for change**
•People do not make good decisions.
•A careful tradeoff study will help you overcome human ineptitude and thereby make better decisions.
04/10/23 © 2009 Bahill
96
Rational decisionsRational decisions**
•One goal
•Perfect information
•The optimal course of action can be described
•This course maximizes expected value
•This is a prescriptive model. We tell people that, in an ideal world, this is how they should make decisions.
04/10/23 © 2009 Bahill
97
SatisficingSatisficing**
•When making decisions there is always uncertainty, too little time and insufficient resources to explore the whole problem space.
•Therefore, people cannot make rational decisions.
•The term satisficing was coined by Noble Laureate Herb Simon in 1955.
•Simon proposed that people do not attempt to find an optimal solution. Instead, they search for alternatives that are good enough, alternatives that satisfice.
04/10/23 © 2009 Bahill
98
04/10/23 © 2009 Bahill
99
Humans are not rationalHumans are not rational**11
•Mark Twain said, “It ain’t what you don’t know that gets you into
trouble. It’s what you know for sure that just ain’t so.”
•Humans are often very certain of knowledge that is false. What American city is directly north of Santiago
Chile? If you travel from Los Angeles to Reno Nevada, in
what direction would you travel? •Most humans think that there are more words that start with the letter r, than there are with r as the third letter.
04/10/23 © 2009 Bahill
100
IllusionsIllusions**
•We call these cognitive illusions.
•We believe them with as much certainty as we believe optical illusions.
04/10/23 © 2009 Bahill
101
The MThe Müüller-Lyer Illusionller-Lyer Illusion**
04/10/23 © 2009 Bahill
102
04/10/23 © 2009 Bahill
103
ObjectiveProbability
SubjectiveProbability
EVRational Behavior V
Subjective Expected Value
Human Behavior
EExpected Utility
Value
Utility
Typical Estimate
0.00.0
1.0
1.0
Ideal Estimate
Ideal Estimate
1.00.00.0
1.0
Typical Estimate
Subjective Worth
Objective Value
Referencepoint
Gains
Losses
Objective Value
Subjective Worth Gains
LossesReference
point
Real Probability
Real Probability
Su
bje
ctiv
e P
rob
ab
ility
We
igh
ting
Su
bje
ctiv
e P
rob
ab
ility
We
igh
ting
04/10/23 © 2009 Bahill
104
Humans judge probabilities poorlyHumans judge probabilities poorly**
0.00.0
1.0
1.0
Ideal Estimate
Typical Estimate
Real Probability
Su
bje
ctiv
e P
rob
ab
ility
We
igh
ting
04/10/23 © 2009 Bahill
105
Monty Hall ParadoxMonty Hall Paradox11**
04/10/23 © 2009 Bahill
106
Monty Hall ParadoxMonty Hall Paradox22**
04/10/23 © 2009 Bahill
107
Monty Hall ParadoxMonty Hall Paradox33**
04/10/23 © 2009 Bahill
108
Monty Hall ParadoxMonty Hall Paradox44**
04/10/23 © 2009 Bahill
109
Monty Hall ParadoxMonty Hall Paradox55**
•Now here is your problem.
•Are you better off sticking to your original choice or switching?
•A lot of people say it makes no difference.
•There are two boxes and one contains a ten-dollar bill.
•Therefore, your chances of winning are 50/50.
•However, the laws of probability say that you should switch.
Monty Hall knew which door had the donkeyMonty Hall knew which door had the donkey
04/10/23 © 2009 Bahill
110
04/10/23 © 2009 Bahill
111
Monty Hall ParadoxMonty Hall Paradox66**
•The box you originally chose has, and always will have, a one-third probability of containing the ten-dollar bill.
•The other two, combined, have a two-thirds probability of containing the ten-dollar bill.
•But at the moment when I open the empty box, then the other one alone will have a two-thirds probability of containing the ten-dollar bill.
•Therefore, your best strategy is to always switch!
04/10/23 © 2009 Bahill
112
UtilityUtility•We have just discussed the right column, subjective probability.
•Now we will discuss the bottom row, utility Objective
ProbabilitySubjectiveProbability
EVRational Behavior V
Subjective Expected Value
Human Behavior
EExpected Utility
Value
Utility
Typical Estimate
0.00.0
1.0
1.0
Ideal Estimate
Ideal Estimate
1.00.00.0
1.0
Typical Estimate
Subjective Worth
Objective Value
Referencepoint
Gains
Losses
Objective Value
Subjective Worth Gains
LossesReference
point
Real Probability
Real ProbabilityS
ub
ject
ive
Pro
ba
bili
ty W
eig
htin
gS
ub
ject
ive P
rob
ab
ility
Weig
htin
g
04/10/23 © 2009 Bahill
113
UtilityUtility•Utility is a measure of the happiness, satisfaction or reward a person gains (or loses) from receiving a good or service.
•Utilities are numbers that express relative preferences using a particular set of assumptions and methods.
•Utilities include both subjectively judged value and the assessor's attitude toward risk.
04/10/23 © 2009 Bahill
114
RiskRisk•Systems engineers use risk to evaluate and manage bad things that could happen, hazards. Risk is measured with the frequency (or probability) of occurrence times the severity of the consequences.
•However, in economics and in the psychology of decision making, risk is defined as the variance of the expected value, uncertainty.*
p1 x1 p2 x2 Risk, uncertaint
y
A 1.0 $10 $10 $0 none
B 0.5 $5 0.5 $15 $10 $25 medium
C 0.5 $1 0.5 $19 $10 $81 high
2
04/10/23 © 2009 Bahill
115
Ambiguity, uncertainty and hazards*Ambiguity, uncertainty and hazards*•Hazard: Would you prefer my forest picked mushrooms or portabella mushrooms from the grocery store?
•Uncertainty: Would you prefer one of my wines or a Kendall-Jackson Napa Valley merlot?
•Ambiguity: Would you prefer my saffron and oyster sauce or marinara sauce?
04/10/23 © 2009 Bahill
116
Gains and losses are not valued equallyGains and losses are not valued equally**
Gains
Losses
ObjectiveValue
Reference Point
SubjectiveWorth
04/10/23 © 2009 Bahill
117
Humans are not rationalHumans are not rational22
•Even if they had the knowledge and resources, people would not make rational decisions, because they do not evaluate utility rationally.
•Most people would be more concerned with a large potential loss than with a large potential gain. Losses are felt more strongly than equal gains.
•Which of these wagers would you prefer to take?*
$2 with probability of 0.5 and $0 with probability 0.5
$1 with probability of 0.99 and $1,000,000 with probability 0.00000001
$3 with probability of 0.999999 and -$1,999,997 with probability 0.000001
04/10/23 © 2009 Bahill
118
Humans are not rationalHumans are not rational33
$2 with probability of 0.5 or $0 with probability 0.5
$0
04/10/23 © 2009 Bahill
119
Humans are not rationalHumans are not rational44
$1 with probability of 0.99
$1,000,000 with probability 0.00000001
04/10/23 © 2009 Bahill
120
Humans are not rationalHumans are not rational55
You owe me two million
dollars!
$3 with probabilityof 0.999999
-$1,999,997 with probability 0.000001
04/10/23 © 2009 Bahill
121
Humans are not rationalHumans are not rational66
•Which of these wagers would you prefer to take?
$2 with probability of 0.5 or $0 with probability 0.5
$1 with probability of 0.99 or $1,000,000 with probability 0.00000001
$3 with probability of 0.999999 or -$1,999,997 with probability 0.000001
•Most engineers prefer the $2 bet•Very few people choose the $3 bet
•All three have an expected value of $1
04/10/23 © 2009 Bahill
122
Subjective expected utilitySubjective expected utilitycombines two subjective concepts: utility and probability.
•Utility is a measure of the happiness or satisfaction a person gains from receiving a good or service.
•Subjective probability is the person’s assessment of the frequency or likelihood of the event occurring.
•The subjective expected utility is the product of the utility times the probability.
04/10/23 © 2009 Bahill
123
Subjective expected utility theorySubjective expected utility theorymodels human decision making as maximizing
subjective expected utility maximizing, because people choose the set of
alternatives with the highest total utility, subjective, because the choice depends on the
decision maker’s values and preferences, not on reality (e.g. advertising improves subjective perceptions of a product without improving the product), and expected, because the expected value is used.
• This is a first-order model for human decision making.
• Sometimes it is called Prospect Theory*.
04/10/23 © 2009 Bahill
124
ObjectiveProbability
SubjectiveProbability
EVRational Behavior V
Subjective Expected Value
Human Behavior
EExpected Utility
Value
Utility
Typical Estimate
0.00.0
1.0
1.0
Ideal Estimate
Ideal Estimate
1.00.00.0
1.0
Typical Estimate
Subjective Worth
Objective Value
Referencepoint
Gains
Losses
Objective Value
Subjective Worth Gains
LossesReference
point
Real Probability
Real Probability
Su
bje
ctiv
e P
rob
ab
ility
We
igh
ting
Su
bje
ctiv
e P
rob
ab
ility
We
igh
ting
04/10/23 © 2009 Bahill
125
Why teach tradeoff studies?Why teach tradeoff studies?•Because emotions, cognitive illusions, biases, fallacies, fear of regret and use of heuristics make humans far from ideal decision makers.
•Using tradeoff studies judiciously can help you make rational decisions.
•We would like to help you move your decisions from the normal human decision-making lower-right quadrant to the ideal decision-making upper-left quadrant.
04/10/23 © 2009 Bahill
126
Components of a tradeoff studyComponents of a tradeoff study Problem statement•Evaluation criteria
•Weights of importance
•Alternative solutions
•Evaluation data
•Scoring functions
•Normalized scores
•Combining functions
•Preferred alternatives
•Sensitivity analysis
04/10/23 © 2009 Bahill
127
Problem statementProblem statement•Stating the problem properly is one of the systems engineer’s most important tasks, because an elegant solution to the wrong problem is less than worthless.
•Problem stating is more important than problem solving.
•The problem statement describes the customer’s needs, states the goals of the project, delineates the scope of the problem, reports the concept of operations, describes the stakeholders, lists the deliverables and presents the key decisions that must be made.
04/10/23 © 2009 Bahill
128
Components of a tradeoff studyComponents of a tradeoff study•Problem statement
Evaluation criteria•Weights of importance
•Alternative solutions
•Evaluation data
•Scoring functions
•Scores
•Combining functions
•Preferred alternatives
•Sensitivity analysis
04/10/23 © 2009 Bahill
129
Evaluation criteriaEvaluation criteria•are derived from high priority tradeoff requirements.
•should be independent, but show compensation.
•Each alternative will be given a value that indicates the degree to which it satisfies each criterion. This should help distinguish between alternatives.
•Evaluation criteria might be things like performance, cost, schedule, risk, security, reliability and maintainability.
04/10/23 © 2009 Bahill
130
Evaluation criterion templateEvaluation criterion template•Name of criterion
•Description
•Weight of importance (priority)
•Basic measure
•Units
•Measurement method
•Input (with expected values or the domain)
•Output
•Scoring function (type and parameters)
•Traces to (requirement of document)
04/10/23 © 2009 Bahill
131
Example criterion packageExample criterion package11
•Name of criterion: Percent Happy Scouts
•Description: The percentage of scouts that leave the race with a generally happy feeling. This criterion was suggested by Sales and Marketing and the Customer.
•Weight of importance: 10
•Basic measure:* Percentage of scouts who leave the event looking happy, contented or pleased
•Units: percentage
•Measurement method: Estimate by the Pinewood Derby Marshall
•Input: The domain is 0 to 100%. The expected values are 70 to 100%.
04/10/23 © 2009 Bahill
132
Example criterion packageExample criterion package22
•Output: 0 to 1
•Scoring function:* Monotonic increasing with lower threshold of 0, baseline of 90, baseline slope of 0.1 and upper threshold of 100.
04/10/23 © 2009 Bahill
133
Second example criterion packageSecond example criterion package11**
•Name of criterion: Total Event Time
•Description: The total event time will be calculated by subtracting the start time from the end time.
•Weight of importance: 8
•Basic measure: Duration of the derby from start to finish.
•Units: Hours
•Measurement method: Observation, recording and calculation by the Pinewood Derby Marshall.
•Input: The domain is 0 to 8 hours. The expected values are 1 to 6 hours.
04/10/23 © 2009 Bahill
134
Second example criterion packageSecond example criterion package22
•Output: 0 to 1
•Scoring function: Biphasic hill shape with lower threshold of 0, lower baseline of 2, lower baseline slope of 0.67, optimum of 3.5, upper baseline of 4.5, upper baseline slope of -1 and upper threshold of 8.
04/10/23 © 2009 Bahill
135
Verboten criteriaVerboten criteria•Availability should not be a criterion, because it cannot be traded off.*
•Assume oranges are available 6 months out of the year.
•Would it make sense to do a tradeoff study selecting between apples and oranges and give oranges an availability expected value of 0.5?
•Suppose your tradeoff study selects oranges, but it is October and oranges are not available: the tradeoff study makes no sense.
04/10/23 © 2009 Bahill
136
Mini-summaryMini-summary
Evaluation criteria are quantitative measures for evaluating how well a system satisfies its performance, cost, schedule or risk requirements.
04/10/23 © 2009 Bahill
137
Evaluation criteria are also called Evaluation criteria are also called • Attributes*• Objectives• Metrics• Measures• Quality characteristics• Figures of merit • Acceptance criteria
“Regardless of what has gone before, the acceptance criteria determine what is actually built.”
04/10/23 © 2009 Bahill
138
Other similar termsOther similar terms• Index • Indicators• Factors• Scales• Measures of Effectiveness • Measures of Performance
04/10/23 © 2009 Bahill
139
MoE versus MoPMoE versus MoP•Generally, it is not worth the effort to debate nuances of these terms. But here is an example.
•Measures of Effectiveness (MoEs) show how well (utility or value) a part of the system mission is satisfied. For an undergraduate student trying to earn a Bachelors degree, his or her class (Freshman, Sophomore, Junior or Senior) would be an MoE.
•Measures of Performance (MoPs) show how well the system functions.For our undergraduate student, their grade point average would be an MoP.*
•MoEs are often computed using several MoPs.
MoEs versus MoPsMoEs versus MoPs22
•The city of Tucson wants to widen Grant Road between I-10 and Alvernon Road. They want six lanes with a median, a 45 mph speed limit, and no traffic jams.
•MoEs cars per day averaged over two weeks cars per hour between 5 and 6 PM, Monday to
Friday, averaged over two weeks•MoPs number of pot holes after one year traffic noise (in dB) at local store fronts smoothness of the surface esthetics of landscaping straightness of the road travel time from I-10 to Alvernon number of traffic lights
04/10/23 © 2009 Bahill
140
MoEs versus MoPsMoEs versus MoPs33
•MoEs are typically owned by the customer
•MoPs are typically owned by the contractor
04/10/23 © 2009 Bahill
141
04/10/23 © 2009 Bahill
142
Moe* thinks at a higher levelthan the mop does
MoEs, MoPs, KPIs, FoMs MoEs, MoPs, KPIs, FoMs and evaluation criteriaand evaluation criteria•MoEs quantify how well the mission is satisfied
•MoPs quantify how well the system functions
•Key performance indices (KPIs) are the most important MoPs
•Evaluation criteria are MoPs that are used in tradeoff studies
•Figures of Merit (FoMs) are the same as evaluation criteria.
•All of these must trace to requirements
04/10/23 © 2009 Bahill
143
04/10/23 © 2009 Bahill
144
Properties of Good Evaluation CriteriaProperties of Good Evaluation Criteria
04/10/23 © 2009 Bahill
145
Properties of good evaluation criteriaProperties of good evaluation criteria• Criteria should be objective• Criteria should be quantitative• Wording of criteria is very important• Criteria should be independent• Criteria should show compensation • Criteria should be linked to requirements • The criteria set should be hierarchical• The criteria set should cover the domain evenly• The criteria set should be transitive• Temporal order should not be important• Criteria should be time invariantOverview slide
04/10/23 © 2009 Bahill
146
Evaluation criteria propertiesEvaluation criteria properties• These properties deal with verification the combining function individual criteria sets of criteria
• But problems created by violating these properties can be ameliorated by reengineering the criteria
04/10/23 © 2009 Bahill
147
Evaluation criteria should be objective Evaluation criteria should be objective (observer independent)(observer independent)• Being Pretty or Nice should not be a criterion
for selecting crewmembers• In sports, Most Valuable Player selections are
often controversial• Deriving a consensus for the Best Football
Player of the Century would be impossible
04/10/23 © 2009 Bahill
148
Evaluation criteria should be quantitativeEvaluation criteria should be quantitativeEach criterion should have a scoring function
04/10/23 © 2009 Bahill
149
Evaluation criteria should be worded in a Evaluation criteria should be worded in a positive manner, so that more is betterpositive manner, so that more is better**
• Use Uptime rather than Downtime.• Use Mean Time Between Failures
rather than Failure Rate.• Use Probability of Success, rather
than Probability of Failure.• When using scoring functions make
sure more output is better
• “Nobody does it like Sara LeeSM”
04/10/23 © 2009 Bahill
150
Exercise: rewrite this statementExercise: rewrite this statementWe have a surgical procedure that should cure your problem. Statistically one percent of the people who undergo this surgery die. Would you like to have this surgery?
04/10/23 © 2009 Bahill
151
Percent happy scoutsPercent happy scouts•The Pinewood Derby tradeoff study had these criteria Percent Happy Scouts Number of Irate Parents
•Because people evaluate losses and gains differently, the Preferred alternatives might have been different if they had used Percent Unhappy Scouts Number of Ecstatic Parents
04/10/23 © 2009 Bahill
152
04/10/23 © 2009 Bahill
153
Criteria should be independentCriteria should be independent• Human Sex and IQ are independent• Human Height and Weight are dependent
04/10/23 © 2009 Bahill
154
The importance of independenceThe importance of independenceBuying a new car, couple-1 criteria• Wife Safety
• Husband Peak Horse Power
04/10/23 © 2009 Bahill
155
Buying a new car, couple-2 criteria Buying a new car, couple-2 criteria •Wife
Safety•Husband
Maximum Horse Power Peak Torque Top Speed Time for the Standing Quarter Mile Engine Size (in liters) Number of Cylinders. Time to Accelerate 0 to 60 mph
What kind of a car do you think they will buy?*
04/10/23 © 2009 Bahill
156
Criteria should show compensationCriteria should show compensationFrom the Systems Engineering literature, tradeoff requirements show compensation
Dictionary definitioncompensate v. 1. To offset: counterbalance.
Compensate means to tradeoff. You are happy to accept less of one thing in order to get more of another and vice versa.
04/10/23 © 2009 Bahill
157
Perfect compensationPerfect compensation• Astronauts growing food on a trip to Mars• Two criteria: Amount of Rice Grown and
Amount of Beans Grown• Goal: maximize* total amount of food• A lot of rice and a few beans is just as good as
lots of beans and little rice• We can tradeoff beans for rice
04/10/23 © 2009 Bahill
158
No compensationNo compensation• A system that produces oxygen and water for
our astronauts
• A system that produced a huge amount of water, but no oxygen might get the highest score, but, clearly, it would not support life for long.
• From Systems Engineering, mandatory requirements show no compensation
04/10/23 © 2009 Bahill
159
Choosing today’s lunchChoosing today’s lunch•Candidate meals: pizza, hamburger, fish & chips, chicken
sandwich, beer, tacos, bread and water•Criteria: Cost, Preparation Time, Tastiness, Novelty, Low
Fat, Contains the Five Food Groups, Complements Merlot Wine, Closeness of Venue
•These criteria are independent and also show compensation
•Criteria are usually nouns, noun phrases or verb phrases
04/10/23 © 2009 Bahill
160
04/10/23 © 2009 Bahill
161
04/10/23 © 2009 Bahill
162
04/10/23 © 2009 Bahill
163
Sometimes it is hard to get both Sometimes it is hard to get both independence and compensationindependence and compensation• If two criteria are independent,
they might not show compensation
• If they show compensation, they might not be independent
• Independence is more important for mandatory requirements
•Compensation is more important for tradeoff requirements
04/10/23 © 2009 Bahill
164
RelationshipsRelationships•Each evaluation criterion must
be linked to a tradeoff requirement. Or in early design phases to a
Mission statement, ConOps, OCD or company policy.
•But only a few tradeoff requirements are used in the tradeoff study.
04/10/23 © 2009 Bahill
165
Evaluation criteria hierarchyEvaluation criteria hierarchy• The criteria tree should be hierarchical• The top level often contains
Performance Cost Schedule Risk
• Dependent entries are grouped into subcategories
• The criteria set should cover the domain evenly
04/10/23 © 2009 Bahill
166
Evaluation criteria set should be transitiveEvaluation criteria set should be transitive**
If A is preferred to B,and B is preferred to C,then A should be preferred to C.
This property is needed for assigning weights.
04/10/23 © 2009 Bahill
167
Temporal order Temporal order should not be importantshould not be important Criteria should be created so that the temporal order is not important for verifying or combining.
04/10/23 © 2009 Bahill
168
The temporal order of verifying The temporal order of verifying criteria should not be important criteria should not be important •Criteria requiring that clothing be Flame Proof
and Water Resistant would make the verification results depend on which we tested first
If the criteria depend on temporal order, then an expert system or a decision tree might be more suitable
04/10/23 © 2009 Bahill
169
Temporal order Temporal order should not be important should not be important • Fragment of a job application
• Q: “Have you ever been arrested?”
A: “No.”• Q: “Why?”
A: “Never got caught.”
04/10/23 © 2009 Bahill
170
The temporal order of combining The temporal order of combining criteria should not be important criteria should not be important • Consider a combining function (CF) that adds
two numbers truncating the fraction(0.2 CF 0.6) CF 0.9 = 0, however,(0.9 CF 0.6) CF 0.2 = 1,the result depends on the order.
• With the Boolean NAND* function ()(0 1) 1 = 0 however, (1 1) 0 = 1, the result depends on the order.
Order of presentation is importantOrder of presentation is important•The stared question is the only question that department and
college promotion committees look at. It is the only question reported in the TCE History.
•Larry Alimony’s CIEQ• I would take another course that was taught this way•The course was quite boring •The instructor seemed interested in students as individuals•The instructor exhibited a through knowledge of the subject matterWhat is your overall rating of this instructor’s teaching
effectiveness?
•TCE What is your overall rating of this instructor’s teaching
effectiveness?•What is your overall rating of the course?•Rate the usefulness of HW, projects, etc. •What is your rating of this instructor compared to other
instructors?•The difficulty level of the course is …
04/10/23 © 2009 Bahill
171
04/10/23 © 2009 Bahill
172
Criteria should be time invariantCriteria should be time invariant•Criteria should not change with
time
• It would be nice if the evaluation data also did not change with time, but this is unrealistic
04/10/23 © 2009 Bahill
173
Evaluation cEvaluation criteria libraryriteria library•Criteria should be created so that they can be reused.
•Your company should have library of generic criteria.•Each criterion package would have the following slots Name DescriptionWeight of importance (priority) Basic measure UnitsMeasurement method Input (with allowed and expected range) Output Scoring function (type and parameters) Trace to (document)
04/10/23 © 2009 Bahill
174
Components of a tradeoff studyComponents of a tradeoff study•Problem statement
•Evaluation criteria
Weights of importance•Alternative solutions
•Evaluation data
•Scoring functions
•Scores
•Combining functions
•Preferred alternatives
•Sensitivity analysis
04/10/23 © 2009 Bahill
175
Weights of importanceWeights of importanceThe decision maker should assign weights so that the more important criteria will have more effect on the outcome.
04/10/23 © 2009 Bahill
176
Using weightsUsing weights
For the Sum Combining Function
For the Product Combining Function, the weights should be put in the exponent
j
j1
weightOutput scoren
j
j
j1
weightOutput scoren
j
j j1
Output weight scoren
j
j j1
Output weight scoren
j
04/10/23 © 2009 Bahill
177
Part of a Pinewood Derby tradeoff studyPart of a Pinewood Derby tradeoff studyPerformance figures of merit evaluated on a prototype for a Round Robin with Best Time Scoring Figure of Merit Input
value Score Weight Score
times weight
1. Average Races per Car
6 0.94 0.20 0.19
2. Number of Ties 0 1 0.20 0.20 3. Happiness 0.87 0.60 0.52 Qualitative
weight Normalized weight
Input value
Scoring function
Score Score times weight
3.1 Percent Happy Scouts
10 0.50 96 0.98
96
0.98 0.49
3.2 Number of Irate Parents
5 0.25 1
1
0.5
1
0.5
0.50 0.13
3.3 Number of Lane Repeats
5 0.25 0 1.0
0
1.00 0.25
Sum 0.87 0.91
04/10/23 © 2009 Bahill
178
Aspects that help establish weightsAspects that help establish weights
Reference: A Prioritization Process
Organizational Commitment Time Required Criticality to Mission Success Risk Architecture Safety Business Value Complexity Priority of Scenarios (use cases) Implementation
Difficulty Frequency of Use Stability Benefit Dependencies Cost Reuse Potential Benefit to Cost Ratio When it is needed
04/10/23 © 2009 Bahill
179
04/10/23 © 2009 Bahill
180
Cardinal versus ordinalCardinal versus ordinal•Weights should be cardinal measures not ordinal measures.
•Cardinal measures indicate size or quantity.
•Ordinal measures merely indicate rank ordering.*
•Cardinal numbers do not just tell us that one criteria is more important than another – they tell us how much more important.
•If one criteria has a weight of 6 and another a weight of 3, then the first is twice as important as the second.
04/10/23 © 2009 Bahill
181
Methods for deriving weights*Methods for deriving weights*• Decision maker assigns numbers between 1 and 10 to
criteria*
• Decision maker rank orders the criteria*
• Decision maker makes pair-wise comparisons of criteria (AHP)*
• Algorithms are available that combine performance, cost, schedule and risk
• Quality Function Deployment (QFD)
• The method of swing weights
• Some people advocate assigning weights only after deriving evaluation data*
04/10/23 © 2009 Bahill
182
Components of a tradeoff studyComponents of a tradeoff study•Problem statement
•Evaluation criteria
•Weights of importance
Alternative solutions•Evaluation data
•Scoring functions
•Scores
•Combining functions
•Preferred alternatives
•Sensitivity analysis
04/10/23 © 2009 Bahill
183
AlternativesAlternatives
04/10/23 © 2009 Bahill
184
The Do Nothing AlternativeThe Do Nothing Alternative
04/10/23 © 2009 Bahill
185
The status quoThe status quo"Selecting an option from a group of similar options can be difficult to justify and thus may increase the apparent attractiveness of retaining the status quo. To avoid this tendency, the decision maker should identify each potentially attractive option and compare it directly to the status quo, in the absence of competing alternatives. If such direct comparison yields discrepant judgments, the decision maker should reflect on the inconsistency before making a final choice."
Redelmeier and Shafir, 1995
04/10/23 © 2009 Bahill
186
Selecting a new carSelecting a new carBahill has a Datsun 240Z with 160,000 miles
His replacement options are
DoDoNothingNothing
04/10/23 © 2009 Bahill
187
The Do Nothing alternatives forThe Do Nothing alternatives forreplacing a Datsun 240Z Status quo, keep the 240Z Nihilism, do without a car, i.e., walk or take
the bus
04/10/23 © 2009 Bahill
188
If the Do Nothing alternative wins,If the Do Nothing alternative wins,your Cost, Schedule and Risk criteria may have overwhelmed your Performance criteria.
04/10/23 © 2009 Bahill
189
If a Do Nothing alternative winsIf a Do Nothing alternative wins22
• Just as you should not add apples and oranges, you should not combine Performance, Cost, Schedule and Risk criteria with each other Combine the Performance criteria (with their
weights normalized so that they add up to one)
Combine the Cost criteria Combine the Schedule criteria Combine the Risk criteria
•Then the Performance, Cost, Schedule and Risk combinations can be combined with clearly stated weights, 1/4, 1/4, 1/4 and 1/4 could be the default.
• If a Do Nothing alternative still wins, you may have the weight for Performance too low.
04/10/23 © 2009 Bahill
190
Balanced scorecardBalanced scorecardThe Business community says that
you should balance these perspectives: Innovation (Learning and Growth) Internal Processes Customer Financial
04/10/23 © 2009 Bahill
191
Sacred cowsSacred cows**
• One important purpose for including a do nothing alternative (and other bizarre alternatives) is to help get the requirements right. If a bizarre alternative wins the tradeoff analysis, then you do not have the requirements right.
• Similarly including sacred cows in the alternatives, will also test the adequacy of the requirements.
• “For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.” -- Richard Feynman
04/10/23 © 2009 Bahill
192
Alternative conceptsAlternative concepts• When formulating alternative concepts,
remember Miller’s* “magical number seven, plus or minus two.”
• Also remember that introducing more alternatives only confuses the DM and makes him or her less likely to choose one of the new alternatives.**
04/10/23 © 2009 Bahill
193
SynonymsSynonyms•Alternative concepts
•Alternative solutions
•Alternative designs
•Alternative architectures
•Options
04/10/23 © 2009 Bahill
194
RiskRisk•The risks included in a tradeoff study
should only be those that can be traded-off. Do not include the highest-level risks.
•Risks might be computed in a separate section, because they usually use the product combining function.
04/10/23 © 2009 Bahill
195
CAIVCAIV•Cost as an independent variable (CAIV)
•Treating CAIV means that you should do the tradeoff study with a specific cost and then go talk to your customer and see what performance, schedule and risk requirements he or she is willing to give up in order to get that cost.
•So if you want to treat CAIV, then keep your tradeoff study independent of cost: that is, do not use cost criteria in your tradeoff study.
04/10/23 © 2009 Bahill
196
Two types of requirementsTwo types of requirements•There are two types of requirementsmandatory requirements tradeoff requirements
04/10/23 © 2009 Bahill
197
Mandatory requirementsMandatory requirements•Mandatory requirements specify necessary and sufficient capabilities that the system must have to satisfy customer needs and expectations.
•They use the words shall or must.
•They are either passed or failed, with no in between.
•They should not be included in a tradeoff study.
•Here is an example of a mandatory requirement: The system shall not violate federal, state or
local laws.
04/10/23 © 2009 Bahill
198
Tradeoff requirementsTradeoff requirements•Tradeoff requirements state capabilities that would make the customer happier.
•They use the words should or want. •They use measures of effectiveness and scoring functions.
•They are evaluated with multicriterion decision techniques.
•There will be tradeoffs among these requirements. •Here is an example of a tradeoff requirement: Dinner should have items from each of the five food groups: Grains, Vegetables, Fruits, Wine, Milk , and Meat and Beans.
•Mandatory requirements are often the upper or lower limits of tradeoff requirements.
04/10/23 © 2009 Bahill
199
Mandatory requirementsMandatory requirementsshould not be in a tradeoff study, because they cannot be traded off.
•Including them screws things up incredibly.
04/10/23 © 2009 Bahill
200
Components of a tradeoff studyComponents of a tradeoff study•Problem statement
•Evaluation criteria
•Weights of importance
•Alternative solutions
Evaluation data•Scoring functions
•Scores
•Combining functions
•Preferred alternatives
•Sensitivity analysis
04/10/23 © 2009 Bahill
201
Evaluation dataEvaluation data11
•Evaluation data come from approximations, product literature, analysis, models, simulations, experiments and prototypes.
•It would be nice if these values were objective, but sometimes we must resort to elicitation of personal preferences.*
•They will be measured in natural units.
04/10/23 © 2009 Bahill
202
Evaluation dataEvaluation data22
•Evaluation data should be entered into the matrix one row (one criterion) at a time.
•They indicate the degree to which each alternative satisfies each criterion.
•They are not probabilities: they are more like fuzzy numbers, degree of membership or degree of fulfillment.
04/10/23 © 2009 Bahill
203
UncertaintyUncertainty•Evaluation data (and weights of importance) should, when convenient, have measures of uncertainty associated with the data.
•This could be done with probability density functions, fuzzy numbers, variance, expected range, certainty factors, confidence intervals, or simple color coding.
04/10/23 © 2009 Bahill
204
NormalizationNormalization**
•Evaluation data are transformed into normalized scores by scoring functions (utility curves) or qualitative scales (fuzzy sets).
•The outputs of such transformations should be cardinal numbers representing the DMs utility.
04/10/23 © 2009 Bahill
205
Scoring function exampleScoring function exampleThis scoring function reflects the DM’s utility that he would be twice as satisfied if there were 91% happy scouts compared to 88% happy scouts.*
04/10/23 © 2009 Bahill
206
QualitativeQualitative scales examples scales examplesEvaluation data Qualitative
evaluationOutput
Good example
0 to 86% happy scouts Not satisfied 0.2
86 to 89% happy scouts Marginally satisfied 0.4
89 to 91% happy scouts Satisfied 0.6
91 to 93% happy scouts Very satisfied 0.8
93 to 100% happy scouts
Elated 1.0
Bad example
0 to 20% happy scouts Not satisfied 0.2
20 to 40% happy scouts Marginally satisfied 0.4
40 to 60% happy scouts Satisfied 0.6
60 to 80 % happy scouts Very satisfied 0.8
80 to 100% happy scouts
Elated 1.0
04/10/23 © 2009 Bahill
207
Components of a tradeoff studyComponents of a tradeoff study•Problem statement
•Evaluation criteria
•Weights of importance
•Alternative solutions
•Evaluation data
Scoring functions•Scores
•Combining functions
•Preferred alternatives
•Sensitivity analysis
04/10/23 © 2009 Bahill
208
What is the best package of soda pop to buy?*What is the best package of soda pop to buy?*Regular price of Coca-Cola in Tucson, January 1995.The Cost criterion is the reciprocal of price.The Performance criterion is the quantity in liters.
Choosing Amongst Alternative Soda Pop Packages Data Criteria Trade-off Values
Item Price (dollars)
Cost (dollars-1)
Quantity (liters)
Sum Product Sum Minus
Product
Com-promise with p=2
Com-promise
with p=10 1 can 0.50 2.00 0.35 2.35 0.70 1.65 2.03 2.00 20 oz 0.60 1.67 0.59 2.26 0.98 1.27 1.77 1.67 1 liter 0.79 1.27 1.00 2.27 1.27 1.00 1.62 1.27 2 liter 1.29 0.78 2.00 2.78 1.56 1.22 2.15 2.00 6 pack 2.29 0.44 2.13 2.57 0.94 1.63 2.17 2.13 3 liter 1.69 0.59 3.00 3.59 1.78 1.81 3.06 3.00 12 pack 3.59 0.28 4.26 4.54 1.19 3.35 4.27 4.26 24 pack 5.19 0.19 8.52 8.71 1.62 7.09 8.52 8.52
Choosing Amongst Alternative Soda Pop Packages Data Criteria Trade-off Values
Item Price (dollars)
Cost (dollars-1)
Quantity (liters)
Sum Product Sum Minus
Product
Com-promise with p=2
Com-promise
with p=10 1 can 0.50 2.00 0.35 2.35 0.70 1.65 2.03 2.00 20 oz 0.60 1.67 0.59 2.26 0.98 1.27 1.77 1.67 1 liter 0.79 1.27 1.00 2.27 1.27 1.00 1.62 1.27 2 liter 1.29 0.78 2.00 2.78 1.56 1.22 2.15 2.00 6 pack 2.29 0.44 2.13 2.57 0.94 1.63 2.17 2.13 3 liter 1.69 0.59 3.00 3.59 1.78 1.81 3.06 3.00 12 pack 3.59 0.28 4.26 4.54 1.19 3.35 4.27 4.26 24 pack 5.19 0.19 8.52 8.71 1.62 7.09 8.52 8.52
04/10/23 © 2009 Bahill
209
Numerical precisionNumerical precision**
04/10/23 © 2009 Bahill
210
The preferred alternative depends on the unitsThe preferred alternative depends on the units
For the Sum but not for the Product Tradeoff Function.
Choosing Amongst Alternative Soda Pop Packages, Effect of Units Item Price
(dollars) Cost
(dollars-1) Quantity (liters)
Sum Product Quantity (barrels)
Sum Product
1 can 0.50 2.00 0.35 2.35 0.70 0.0003 2.0003 0.0060 20 oz 0.60 1.67 0.59 2.26 0.98 0.0050 1.6717 0.0084 1 liter 0.79 1.27 1.00 2.27 1.27 0.0085 1.2785 0.0108 2 liter 1.29 0.78 2.00 2.78 1.56 0.0170 0.7837 0.0132 6 pack 2.29 0.44 2.13 2.57 0.94 0.0181 0.4548 0.0079 3 liter 1.69 0.59 3.00 3.59 1.78 0.0256 0.6173 0.0151 12 pack
3.59 0.28 4.26 4.54 1.19 0.0363 0.3148 0.0101
24 pack
5.19 0.19 8.52 8.71 1.62 0.0726 0.2653 0.0140
Choosing Amongst Alternative Soda Pop Packages, Effect of Units Item Price
(dollars) Cost
(dollars-1) Quantity (liters)
Sum Product Quantity (barrels)
Sum Product
1 can 0.50 2.00 0.35 2.35 0.70 0.0003 2.0003 0.0060 20 oz 0.60 1.67 0.59 2.26 0.98 0.0050 1.6717 0.0084 1 liter 0.79 1.27 1.00 2.27 1.27 0.0085 1.2785 0.0108 2 liter 1.29 0.78 2.00 2.78 1.56 0.0170 0.7837 0.0132 6 pack 2.29 0.44 2.13 2.57 0.94 0.0181 0.4548 0.0079 3 liter 1.69 0.59 3.00 3.59 1.78 0.0256 0.6173 0.0151 12 pack
3.59 0.28 4.26 4.54 1.19 0.0363 0.3148 0.0101
24 pack
5.19 0.19 8.52 8.71 1.62 0.0726 0.2653 0.0140
04/10/23 © 2009 Bahill
211
Scoring functionsScoring functions• Criteria should always have scoring functions so
that the preferred alternatives do not depend on the units used.
• Scoring functions are also called utility functions utility curves value functions normalization functions mappings
04/10/23 © 2009 Bahill
212
Scoring function for CostScoring function for Cost**
04/10/23 © 2009 Bahill
213
Scoring function for QuantityScoring function for Quantity**
A simple program that creates graphs such as these is available for free athttp://www.sie.arizona.edu/sysengr/slides.It is called the Wymorian Scoring Function tool.
04/10/23 © 2009 Bahill
214
The scoring function equationThe scoring function equation**
2×Slope× Baseline+CriteriaValue-2×Lower
1SSF1
Baseline-Lower1
CriteriaValue-Lower
04/10/23 © 2009 Bahill
215
Evaluation data may be logarithmicEvaluation data may be logarithmic**
04/10/23 © 2009 Bahill
216
The need for scoring functionsThe need for scoring functions11**
•You can add $s and £s, but
•you can’t add $s and lbs.
04/10/23 © 2009 Bahill
217
The need for scoring functionsThe need for scoring functions22
•Would you add values for something that cost a billion dollars and lasted a nanosecond?*
•Alt-1 cost a hundred dollars and lasts one millisecond, Sum = 100.001.
•Alt-2 only cost ninety nine dollars but it lasts two millisecond, Sum = 99.002.
•Does the duration have any effect on the decision?
04/10/23 © 2009 Bahill
218
Different Distributions of Alternatives in Different Distributions of Alternatives in Criteria SpaceCriteria Space** May Produce Different May Produce Different
Preferred AlternativesPreferred Alternatives
Tradeoff of requirements*Tradeoff of requirements*
04/10/23 © 2009 Bahill
219
0.25
0.50
0.75
1.00
0.005 10 15 200
Pages per Minute
Cos
t (1
/k$
)4P
4Plus
4Si
04/10/23 © 2009 Bahill
220
Pareto OptimalPareto OptimalMoving from one alternative to another will improve at least one criterion and worsen at least one criterion, i.e., there will be tradeoffs.
“The true value of a service or product is determined by what one is willing to give up to obtain it.”
04/10/23 © 2009 Bahill
221
NomenclatureNomenclature
Real-world data will not fall neatly onto lines such as the circle in the pervious slide. But often they may be bounded by such functions. In the operations research literature such data sets are called convex, although the function bounding them is called concave (Kuhn and Tucker, 1951).
04/10/23 © 2009 Bahill
222
Different distributionsDifferent distributions
The feasible alternatives may have different distributions in the criteria space. These include:
Circle Straight Line Hyperbola
04/10/23 © 2009 Bahill
223
Alternatives on a circleAlternatives on a circle**
Alternatives on a Circle Assume the alternatives are on the circle x2 + y2 = 1
Sum Combining Function: 2x + y = x + 1- x with the derivative
d(Sum Combining Function)/21
x
xdx = 1-
Product Combining Function: 2x× y = x× 1- x with the derivative
d(Product Combining Function)/dx 2
2
2
-x= + 1- x
1- x
Both Combining Functions have maxima at x=y=0.707 (This result does depend on the weights.)
04/10/23 © 2009 Bahill
224
Alternatives on a straight-LineAlternatives on a straight-LineAssume the alternatives are on the straight-line y = -x + 1
Sum Combining Function: x + y = x - x + 1 = 1
All alternatives are optimal (i.e. selection is not possible)
Product Combining Function: x * y = -x2 + x with
d(Product Combining Function)/dx = -2x + 1
Product Combining Function: maximum at x=0.5
Sum Combining Function: all alternatives are equally good
Product Combining Function seems better for decision aiding
04/10/23 © 2009 Bahill
225
Alternatives on a hyperbolaAlternatives on a hyperbola**
Alternatives on a Hyperbola Assume the alternatives are on the hyperbola (x + 1)(y + 1) = 2
Sum Combining Function: x + y = -2
x + 1x +1
with
d(Sum Combining Function)/dx = 2
21-
x +1
Product Combining Function: x * y =2x
- xx +1
with
d(Product Combining Function)/dx = 2
2-1
x +1
04/10/23 © 2009 Bahill
226Figure of Merit 1
Fig
ure
of M
erit
2
1
1
00
Figure of Merit 1
Fig
ure
of M
erit
2
100
Figure of Merit 1
Fig
ure
of M
erit
2
1
1
00
0.71
0.7
0.4
0.4
Sum &Product
ProductProduct
Sum
Sum
0.5
0.5
Sum(all points on line)
Recommended Alternative of
Sum and ProductCombining Functions
Figure of Merit 1
Fig
ure
of M
erit
2
1
1
00
Figure of Merit 1
Fig
ure
of M
erit
2
100
Figure of Merit 1
Fig
ure
of M
erit
2
1
1
00
0.71
0.7
0.4
0.4
Sum &Product
ProductProduct
Sum
Sum
0.5
0.5
Sum(all points on line)
Recommended Alternative of
Sum and ProductCombining Functions
04/10/23 © 2009 Bahill
227Muscle Velocity
Mu
scle
Fo
rce
vmax00
F0Max
Force
Max Speed
Muscle Force-Velocity Relationship
Max Power
A lively baseball debateA lively baseball debate•For over 30 years baseball statisticians have argued over the best measure of offensive effectiveness.
•Two of the most popular measures are On-base plus slugging OPS = OBP + SLG Batter’s run average BRA = OBP x SLG
•I think their arguments ignored the most relevant data, the shape of the distribution of OBP and SLG for major league players.
•If it is circular either will work.
•If it is hyperbolic, do not use the sum.
04/10/23 © 2009 Bahill
228
04/10/23 © 2009 Bahill
229
Muscle force-velocity relationshipMuscle force-velocity relationship• (Force + F0 )(velocity + vmax) = constant, where F0 (the
isometric force) and vmax (the maximum muscle velocity) are constants.
• Humans sometimes use one combining function and sometimes they use another.
• If a bicyclist wants maximum acceleration, he or she uses the point (0, F0). If there is no resistance and maximum speed is desired, use the point (vmax, 0). These solutions result from maximizing the sum of force and velocity.
• However, if there is energy dissipation (e.g., Friction, air resistance) and maximum speed is desired, choose the maximum power point, the maximum product of force and velocity.
• This shows that the appropriate tradeoff function may depend on the task at hand.
04/10/23 © 2009 Bahill
230
Nonconvex data setsNonconvex data setsThe muscle force-velocity relationship fit neatly onto lines such as this hyperbola. This will not always be the case. But when it is not, the data may be bounded by such functions. In the operations research literature such data sets are called concave, although the function bounding them is called convex (Kuhn and Tucker, 1951).
04/10/23 © 2009 Bahill
231
Mini-summaryMini-summary•The Product Combining Function always favors alternatives with moderate scores for all criteria. It rejects alternatives with a low score for any criterion.
•Therefore the Product Combining Function may seem better than the Sum Combining Function. But the Sum Combining Function is used much more in systems engineering.
04/10/23 © 2009 Bahill
232
Components of a tradeoff studyComponents of a tradeoff study•Problem statement
•Evaluation criteria
•Weights of importance
•Alternative solutions
•Evaluation data
•Scoring functions
•Scores
Combining functions•Preferred alternatives
•Sensitivity analysis
04/10/23 © 2009 Bahill
233
Summation is not always Summation is not always the best way to combine datathe best way to combine data**
Hamlet of MontenegroESTABLISHED 2000POPULATION 10ELEVATION 2400TOTAL 4410
04/10/23 © 2009 Bahill
234
Popular combining functionsPopular combining functions• Sum Combining Function = x + y Used most often by engineers
• Product Combining Function = x y Cost to benefit ratio Risk analyses Game theory*
• Sum Minus Product = x + y - xy Probability theory Fuzzy logic systems Expert system certainty factors
• Compromise = 1/pp px + y
04/10/23 © 2009 Bahill
235
XORXOR**
•The previous combining functions implemented an AND function of the criteria.
•There is no combining function that implements the exclusive or (XOR) function, e.g.
•Criterion-1: Fuel consumption in highway driving, miles per gallon of gasoline. Baseline = 23 mpg.
•Criterion-2: Fuel consumption in highway driving, miles per gallon of diesel fuel. Baseline = 26 mpg.
•You want to use criterion-1 for alternatives with gasoline engines and criterion-2 for alternatives with diesel engines.
04/10/23 © 2009 Bahill
236
The American public accepts The American public accepts the Sum Combining Functionthe Sum Combining Function
• It is used to rate NFL quarterbacks
• It is used to select the
best college football teams
04/10/23 © 2009 Bahill
237
NFL quarterback passer ratingsNFL quarterback passer ratings
BM stands for basic measure
BM1 = (Completed Passes) / (Pass Attempts)
BM2 = (Passing Yards) / (Pass Attempts)
BM3 = (Touchdown Passes) / (Pass Attempts)
BM4 = Interceptions / (Pass Attempts)
Rating = [5(BM1-0.3) + 0.25(BM2-3) + 20(BM3) + 25(-BM4+0.095)]*100/6
04/10/23 © 2009 Bahill
238
College football BCSCollege football BCS**
BM1 = Polls: AP media & ESPN coachesBM2 = Computer Rankings: Seattle Times, NY
Times, Jeff Sagarin, etc.BM3 = Strength of ScheduleBM4 = Number of Losses
Rating = [BM1 + BM2 + BM3 - BM4]
http://sports.espn.go.com/ncf/abcsports/BCSStandings
www.bcsFootball.org
04/10/23 © 2009 Bahill
239
What is the best package of soda pop to buy?What is the best package of soda pop to buy?**
Regular price of Coca-Cola in Tucson, January 1995.The Cost criterion is the reciprocal of price.The Performance criterion is the quantity in liters.
Choosing Amongst Alternative Soda Pop Packages Data Criteria Trade-off Values
Item Price (dollars)
Cost (dollars-1)
Quantity (liters)
Sum Product Sum Minus
Product
Com-promise with p=2
Com-promise
with p=10 1 can 0.50 2.00 0.35 2.35 0.70 1.65 2.03 2.00 20 oz 0.60 1.67 0.59 2.26 0.98 1.27 1.77 1.67 1 liter 0.79 1.27 1.00 2.27 1.27 1.00 1.62 1.27 2 liter 1.29 0.78 2.00 2.78 1.56 1.22 2.15 2.00 6 pack 2.29 0.44 2.13 2.57 0.94 1.63 2.17 2.13 3 liter 1.69 0.59 3.00 3.59 1.78 1.81 3.06 3.00 12 pack 3.59 0.28 4.26 4.54 1.19 3.35 4.27 4.26 24 pack 5.19 0.19 8.52 8.71 1.62 7.09 8.52 8.52
Choosing Amongst Alternative Soda Pop Packages Data Criteria Trade-off Values
Item Price (dollars)
Cost (dollars-1)
Quantity (liters)
Sum Product Sum Minus
Product
Com-promise with p=2
Com-promise
with p=10 1 can 0.50 2.00 0.35 2.35 0.70 1.65 2.03 2.00 20 oz 0.60 1.67 0.59 2.26 0.98 1.27 1.77 1.67 1 liter 0.79 1.27 1.00 2.27 1.27 1.00 1.62 1.27 2 liter 1.29 0.78 2.00 2.78 1.56 1.22 2.15 2.00 6 pack 2.29 0.44 2.13 2.57 0.94 1.63 2.17 2.13 3 liter 1.69 0.59 3.00 3.59 1.78 1.81 3.06 3.00 12 pack 3.59 0.28 4.26 4.54 1.19 3.35 4.27 4.26 24 pack 5.19 0.19 8.52 8.71 1.62 7.09 8.52 8.52
04/10/23 © 2009 Bahill
240
ResultsResults• The Product Combining
Function suggests that the preferred package is the three liter bottle
• However, the other combining functions all recommend the 24 pack
• Plotting these data on Cartesian coordinates produces a nonconvex distribution
• The best hyperbolic fit to these data is (quantity + 0.63)(cost + 0.08) = 2
04/10/23 © 2009 Bahill
241
Soda pop dataSoda pop data
0
0.5
1
1.5
2
2.5
0 5 10
Quantity (liters)
Co
st
(1/d
ollars
)
04/10/23 © 2009 Bahill
242
04/10/23 © 2009 Bahill
243
Which matchesWhich matcheshuman decision making?human decision making?•For a nonconvex distribution, the Sum Combining
Function will favor the points at either end of the distribution. Sometimes this matches human decision making. I usually buy a case of soda for my family. A person working in an office building on a
Sunday afternoon might buy a single can from the vending machine.
•A frugal person might want to maximize the product of cost and performance, i.e. the maximum liters/dollar (the biggest bang for the buck), which is the three liter bottle. This matches the recommendation of the Product Combining Function.
04/10/23 © 2009 Bahill
244
Which matches humanWhich matches humandecision making? decision making? (cont.)(cont.)
This example shows that for a nonconvex distribution of alternatives, the choice of the combining function determines the preferred alternative.
04/10/23 © 2009 Bahill
245
Who was the best NFL quarterback?Who was the best NFL quarterback?
• NFL quarterback passer ratings • BM1 = (Completed Passes) / (Pass
Attempts)
• BM2 = (Passing Yards) / (Pass Attempts)
• BM3 = (Touchdown Passes) / (Pass Attempts)
• BM4 = Interceptions / (Pass Attempts)
• Rating = [5(BM1-0.3) + 0.25(BM2-3) + 20(BM3) + 25(-BM4+0.095)]*100/6
04/10/23 © 2009 Bahill
246
The best NFL quarterback for 1999The best NFL quarterback for 1999
http://www.football.espn.go.com/nfl/statistics/
Sum (p=1)
Product Sum Minus Product
Compromise with p=2
Compromise with p=
Kurt Warner
Kurt Warner
Kurt Warner
Kurt Warner
Kurt Warner
Steve Beuerlein
Jeff George
Steve Beuerlein
Steve Beuerlein
Jeff George
Jeff George
Steve Beuerlein
Jeff George
Peyton Manning
Steve Beuerlein
Peyton Manning
Peyton Manning
Peyton Manning
Jeff George
Peyton Manning
Sum (p=1)
Product Sum Minus Product
Compromise with p=2
Compromise with p=
Kurt Warner
Kurt Warner
Kurt Warner
Kurt Warner
Kurt Warner
Steve Beuerlein
Jeff George
Steve Beuerlein
Steve Beuerlein
Jeff George
Jeff George
Steve Beuerlein
Jeff George
Peyton Manning
Steve Beuerlein
Peyton Manning
Peyton Manning
Peyton Manning
Jeff George
Peyton Manning
The best NFL quarterback 1994The best NFL quarterback 1994
04/10/23 © 2009 Bahill
247
Sum Product Sum Minus Product
Compromise with p=
Steve Young Steve Young Steve Bono Steve Bono John Elway John Elway Bubby Brister Steve Young Dan Marino Dan Marino Steve
Beuerlein Bobby Herbert
Bobby Herbert
Bobby Herbert
Jeff George Dan Marino
Eric Kramer Warren Moon Neil O’Donnell Eric Kramer
Sum Product Sum Minus Product
Compromise with p=
Steve Young Steve Young Steve Bono Steve Bono John Elway John Elway Bubby Brister Steve Young Dan Marino Dan Marino Steve
Beuerlein Bobby Herbert
Bobby Herbert
Bobby Herbert
Jeff George Dan Marino
Eric Kramer Warren Moon Neil O’Donnell Eric Kramer
04/10/23 © 2009 Bahill
248
A manned mission to MarsA manned mission to Mars11
•The astronauts will grow beans and rice
•Lots of beans and a little rice is just as good as lots of rice and a few beans
•Both the Sum and the Product Combining Functions work fine
04/10/23 © 2009 Bahill
249
A manned mission to MarsA manned mission to Mars22
•The astronauts need a system that produces oxygen and water
•The Product Combining Function works fine
•But the Sum Combining Function could recommend zero water or zero oxygen
04/10/23 © 2009 Bahill
250
Implementing the combining functionsImplementing the combining functions•The Analytic Hierarchy Process (implemented
by the commercial tool Expert Choice) allows the user to choose between the sum and the product combining functions.
•You would have to implement the other combining functions by yourself.
04/10/23 © 2009 Bahill
251
TheThe compromise combining function*compromise combining function*
Compromise = 1/ pp px y
04/10/23 © 2009 Bahill
252
When should When should pp be 1, 2 or be 1, 2 or ??• Use p = 1 if the criteria show perfect
compensation
• Use p = 2 if you want Euclidean distance.
• Use p = if you are selecting a hero and there is no compensation
• Compromise = 1/ pp px y
04/10/23 © 2009 Bahill
253
If If pp = = •The preferred alternative is the one with the
largest criterion
•There is no compensation, because only one criterion (the largest) is considered
•Compromise Output =
• If p is large and x > y then xp >> yp and
Compromise Output
1/ pp px y
1/ ppx x
04/10/23 © 2009 Bahill
254
Use Use pp = = when selecting when selecting•the greatest athlete of the century using
Number of National Championship Rings* and Peak Salary
•the baseball player of the week using Home Runs and Pitching Strikeouts
•a movie using Romance, Action and Comedy
04/10/23 © 2009 Bahill
255
NBA teams seem to use NBA teams seem to use pp = = • When drafting basketball players
• Criteria are Height and Assists
• They want seven-foot players with ten assists per game (the ideal point)
• In years when there are many point guards but no centers, they draft the best point guards
• Choose the criterion with the maximum score (Assists) and then select the alternative whose number of Assists has the minimum distance to the ideal point
04/10/23 © 2009 Bahill
256
Use Use p p = = when choosing minimax when choosing minimax
• A water treatment plant to reduce the amount of mercury, lead and arsenic in the water.
• Trace amounts are not of concern.
• First, find the poison with the maxmaximum concentration, then choose the alternative with the miniminimum amount of that poison.
• Hence the term minimaxminimax.
04/10/23 © 2009 Bahill
257
Design of a baseball batDesign of a baseball bat• The ball goes the farthest, if it hits the
sweet spot of the bat
• Error = |sweet spot - hit point|
• Loss = number of feet short of 500
• For an amateur use minimax: minimize the Loss, if the Error is maximum
• For Alex Rodriguez use minimin
04/10/23 © 2009 Bahill
258
The distance The distance the ball the ball travels travels
depends on depends on where the ball where the ball
hits the bathits the bat**
SweetSpot
LossError For A-Rod
use minimin
Distance
For Terry use minimax: design the system to minimize the Loss if the Error
is maximum
04/10/23 © 2009 Bahill
259
Use Use pp = = if you are very risk averseif you are very risk averse•A million dollar house on a river bank: a 100-year
flood would cause $900K damage•A million dollar house on a mountain top: a violent
thunderstorm would cause $100K damage•Minimax: choose the worst risk, the 100-year
flood, and choose the alternative that minimizes it: build your house on the mountain top*
04/10/23 © 2009 Bahill
260
Use Use pp = 1 if you are probabilistic = 1 if you are probabilistic**
•Risk equals (probability times severity of a 100 year flood) plus (probability times severity of a violent thunderstorm)
•Risk(River Bank) = 0.01×0.9 + 0.1×0 = 0.009
•Risk(Mountain Top) = 0.01×0 + 0.1×0.1 = 0.010
•Therefore, build your house on the river bank
04/10/23 © 2009 Bahill
261
SynonymsSynonyms•Combining functions are also called objective functions optimization functions performance indices
•Combining functions may include probability density functions*
04/10/23 © 2009 Bahill
262
Summary about combining functionsSummary about combining functions•Summation of weighted scores is the most common.
•Product combining function eliminates alternatives with a zero for any criterion.*
•Compromise function with p=∞ uses only one criterion.
04/10/23 © 2009 Bahill
263
Components of a tradeoff studyComponents of a tradeoff study•Problem statement
•Evaluation criteria
•Weights of importance
•Alternative solutions
•Evaluation data
•Scoring functions
•Scores
•Combining functions
Preferred alternatives•Sensitivity analysis
04/10/23 © 2009 Bahill
264
Select preferred alternativesSelect preferred alternatives• Select the preferred alternatives.
• Present the results of the tradeoff study to the original decision maker and other relevant stakeholders.
• A sensitivity analysis will help validate your study.
04/10/23 © 2009 Bahill
265
SynonymsSynonyms•Preferred alternatives
•Recommended alternatives
•Preferred solutions
04/10/23 © 2009 Bahill
266
Components of a tradeoff studyComponents of a tradeoff study•Problem statement
•Evaluation criteria
•Weights of importance
•Alternative solutions
•Evaluation data
•Scoring functions
•Scores
•Combining functions
•Preferred alternatives
Sensitivity analysis
04/10/23 © 2009 Bahill
267
PurposePurposeA sensitivity analysis identifies the most important parameters in a tradeoff study.
04/10/23 © 2009 Bahill
268
Sensitivity analysesSensitivity analyses•A sensitivity analysis of the tradeoff study is
imperative.
•Vary the inputs and parameters and discover which ones are the most important.
•The Pinewood Derby had 89 criteria. Only three of them could change the preferred alternative.
04/10/23 © 2009 Bahill
269
Sensitivity analysis of Pinewood Derby (simulation data)Sensitivity analysis of Pinewood Derby (simulation data)
Sensitivity Analysis of Pinewood Derby (simulation data)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Performance Weight
Ove
rall
Sco
re
Single eliminationDouble eliminationRound robin, mean-timeRound robin, best-time Round robin, points
04/10/23 © 2009 Bahill
270
The Do Nothing alternativesThe Do Nothing alternatives• The double elimination tournament was the
status quo.
• The single elimination tournament was the nihilistic do nothing alternative.
04/10/23 © 2009 Bahill
271
Sensitivity analysis of Pinewood Derby (prototype data)Sensitivity analysis of Pinewood Derby (prototype data)
Sensitivity of Pinewood Derby (prototype data)
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Performance Weight
Ove
rall
Sco
re
Double eliminationRound robin, best-time Round robin, points
04/10/23 © 2009 Bahill
272
Semirelative-sensitivity functionsSemirelative-sensitivity functions
The semirelative-sensitivity of the function F to variations in the parameter is
0NOP
F FS
04/10/23 © 2009 Bahill
273
Tradeoff studyTradeoff studyA Generic Tradeoff Study
Criteria Weight of
Importance Alternative
1 Alternative
2 Criterion 1 Wt1 S11 S12 Criterion 2 Wt2 S21 S22 Final Score F1 F2
A Numeric Example of a Tradeoff Study Alternatives
Criteria Weight of
Importance Umpire’s Assistant
Seeing Eye Dog
Accuracy 0.75 0.67 0.33 Silence of Signaling
0.25 0.83 0.17
Sum of weight times score
0.71 The
winner 0.29
1 1 11 2 21 2 1 12 2 22andF Wt S Wt S F Wt S Wt S
04/10/23 © 2009 Bahill
274
Which parameters could change Which parameters could change the recommendations?the recommendations?Use this performance index*
Compute the semirelative-sensitivity functions.
1 2 1 11 2 21 1 12 2 22 0.420F F F Wt S Wt S Wt S Wt S
04/10/23 © 2009 Bahill
275
Semirelative-sensitivity functions*Semirelative-sensitivity functions*
1
2
11
21
12
22
11 12 1
21 22 2
1 11
2 21
1 12
2 22
0.26
0.16
0.50
0.21
-0.25
-0.04
FWt
FWt
FS
FS
FS
FS
S S S Wt
S S S Wt
S Wt S
S Wt S
S Wt S
S Wt S
04/10/23 © 2009 Bahill
276
What about interactions?What about interactions?The semirelative-sensitivity function for the interaction of Wt1 and S11 is
which is bigger than the first-order terms.
1 11 0 0 0 0
2
1 11 1 111 11 NOP
0.5025FWt S
FS Wt S Wt S
Wt S
04/10/23 © 2009 Bahill
277
InteractionsInteractions
So interactions are important.
Semirelative Sensitivity Values Showing Interaction Effects
Function Nominal values
Values increased by 10%
New F values F
Total change in z
1
FWtS 1Wt =0.75 1Wt =0.82 0.446 0.026
11
FSS 11S =0.67 11S =0.74 0.470 0.050
0.076F
1 11
FWt SS 1Wt =0.75
11S =0.67 1Wt =0.82
11S =0.74 0.501 0.081 0.081
04/10/23 © 2009 Bahill
278
Estimating derivativesEstimating derivatives
If (x-x0) and f” are small, then the second term on the right can be neglected.
20 0 0 0
( )( ) ( ) ( )( ) ( )
2!
ff x f x f x x x x x
04/10/23 © 2009 Bahill
279
Tradeoff study exampleTradeoff study exampleFor a +5% parameter change the semirelative-sensitivity function is
This is very easy to compute.
0 00
200.05
F F FS F
1120(0.025) 0.5F
SS
Tradeoff Study with S11 Increased by 5%
Criteria Weight of
Importance Umpire’s Assistant
Seeing Eye dog
Accuracy 0.75 0.70 0.33 Silence of Signaling 0.25 0.83 0.17 Sum of weight times score
0.74 0.29
04/10/23 © 2009 Bahill
280
Estimated semirelative sensitivitiesEstimated semirelative sensitivities
This is the same result that we
previouslyobtained
analytically.
The Semirelative Sensitivity of the Difference Between the Two Output Scores computed with a Plus 5% Parameter Perturbation
Function Value
1
FWtS +0.26
2
FWtS +0.16
11
FSS +0.50
21
FSS +0.21
12
FSS -0.25
22
FSS -0.04
04/10/23 © 2009 Bahill
281
But what about the second-order terms? But what about the second-order terms? Namely
When using the sum of weighted scores combining function
the second derivatives are all zero. So our estimations are all right. This is not true for the product combining function
or most other common combining functions. See Daniels, Werner and Bahill [2001] for explanations of other combining functions.
20
( )( )
2!
fx x
1 1 11 2 21 2 1 12 2 22andF Wt S Wt S F Wt S Wt S
1 2 1 21 11 21 2 12 22 and Wt Wt Wt WtF S S F S S
04/10/23 © 2009 Bahill
282
The moral of this storyThe moral of this storyThe perturbation step size (x – x0) should be small. Five and ten percent step sizes are probably too big, but we have been getting away with it, because we usually use the sum combining function.
04/10/23 © 2009 Bahill
283
Derivative of a function of two variablesDerivative of a function of two variables
•Let us examine the second-order terms,
those inside the { }, for two reasons to see if they are large and must be included in computing the first derivative to estimate the effects of interactions on the sensitivity analysis
0 0 0 0 0 0 0 0
2 20 0 0 0
( , ) ( , ) ( , )( ) ( , )( )
( , )( ) 2 ( , )( )( ) ( , )( )
x y
xx xy yy
f x y f x y f x y x x f x y y y
f x x f x x y y f y y
04/10/23 © 2009 Bahill
284
InteractionsInteractionsPreviously we derived the analytic semirelative-sensitivity function for the interaction of Wt1 and S11 as,
which is bigger than the first-order semirelative-sensitivity functions.
1 11 0 0 0 0
2
1 11 1 111 11 NOP
0.5025FWt S
FS Wt S Wt S
Wt S
04/10/23 © 2009 Bahill
285
InteractionsInteractionsFor a 5% change in parameter values, a simple-minded approximation is
using our tradeoff study values we get
This does not match the analytic value.
What went wrong?
2
2
0 0 0 00 0
200.05 0.05
F FF FS F
1 11
220 0.6125F
Wt SS F
04/10/23 © 2009 Bahill
286
How big are the second-order terms?How big are the second-order terms?
In estimating
the sum of the first order-terms is 0.00038
the sum of second order terms is 0.00123.
The second-order terms cannot be ignored.
1 12
FWt SS
04/10/23 © 2009 Bahill
287
Step sizeStep sizeCan we fix this problem by using a smaller step size?
If we reduce the step size to 0.1%
This still does not match the analytic result.
2
2
0 0 0 00 0
10000.001 0.001
F FF FS F
1 11
21000 0.5746F
Wt SS F
04/10/23 © 2009 Bahill
288
It’s not the step sizeIt’s not the step sizeBut this time the fault is not that of too large of a step size, because in estimating
the sum of the first order-terms is 0.000757 and
the sum of second order terms is 0.000001.
The second order terms can be ignored.
1 11
FWt SS
04/10/23 © 2009 Bahill
289
What went wrong?What went wrong?In the previous computations, we changed both parameters at the same time and then compared the value of the function to the value of the function at its normal operating point. However, this is not the correct estimation for the second-partial derivative.
04/10/23 © 2009 Bahill
290
Estimating the second partialsEstimating the second partials11
To estimate the second-partial derivatives we should start with
20 0 0 0 0( , ) ( , ) ( , )f f f
0 0 0 02
0 0
( , ) ( , ) ( , ) ( , )( , )
f f f ff
20 0 0 0 0 0( , ) ( , ) ( , ) ( , ) ( , )f f f f f
04/10/23 © 2009 Bahill
291
Estimating the second partialsEstimating the second partials22
21 11
1 11
( , ) 0.4207580 0.4205025 0.4202550 0.42000001
0.00075*0.00067
f Wt S
Wt S
Values to be Used in Estimating the Second Derivative
Terms Parameter values with a 0.1% step size, that is 1Wt =0.00075 and 11S =0.00067
Function values
( , )f 1Wt =0.75075
11S =0.67067 0.4207580
0( , )f 11S =0.67067 0.4205025
0( , )f 1Wt =0.75075 0.4202550
0 0( , )f 1Wt =0.75000
11S =0.67000 0.4200000
04/10/23 © 2009 Bahill
292
Estimating the sensitivity functionsEstimating the sensitivity functionsTo get the semirelative-sensitivity function we multiply the second-partial derivative by the normal values of Wt1 and S11 to get
Now, this is the same result that we derived in the analytic semirelative-sensitivity section.
1 11 0 0 0 0
21 11
1 11 1 111 11 NOP
( , )1 0.5025F
Wt S
f Wt SS Wt S Wt S
Wt S
04/10/23 © 2009 Bahill
293
Lessons learnedLessons learned•The perturbation step size should be small. Five and 10% perturbations are not acceptable.
•It is incorrect to estimate the second partial derivative by changing two parameters at the same time and then comparing that value of the function to the value of the function at its normal operating point. Estimating second derivatives requires evaluation of four not two numerator terms.
04/10/23 © 2009 Bahill
294
Other Techniques for Combining Data in Other Techniques for Combining Data in Order to Find the Preferred alternativesOrder to Find the Preferred alternatives
04/10/23 © 2009 Bahill
295
The Ideal PointThe Ideal Point11
•The ideal point is the point where all the criteria have their optimal scores.
• In the soda pop example we will define the ideal point as the intercepts of the hyperbola fit to the data.
04/10/23 © 2009 Bahill
296
The Ideal PointThe Ideal Point22
The preferred alternative is found by minimizing the distance to the ideal point using LP metrics.
where zk is the score of the kth criterion, wk is the weight of the kth criterion, z*k is the kth component of the ideal point, z*k is the kth component of the anti-ideal point and n is the number of criteria. The criteria index is k and the alternatives index is i.
1
1
n pp p
p k kk
L w d
*
**
-
-
k k
k
k k
z zd
z z
04/10/23 © 2009 Bahill
297
The Ideal PointThe Ideal Point33
Our modified Minkowski metrics:
1
np p
p k kk
L w d
04/10/23 © 2009 Bahill
298
Ideal PointIdeal Point44**
The Ideal Point
0
1
2
3
0 10 20
Quantity (liters)
Co
st
(1/d
olla
rs)
d i
Idea l P o in t
04/10/23 © 2009 Bahill
299
The Ideal PointThe Ideal Point55**
Using wi = 1 and p equal to 1, 2, and we get
Using the Ideal Point to Select Soda Pop Packages Data Criteria Trade-off Values
Item Price (dollars)
Cost (dollars-1)
Quantity (liters)
L1 norm
L2 norm
L norm
1 can 0.50 2.00 0.35 1.34 1.04 0.986 20 oz 0.60 1.67 0.59 1.44 1.07 0.976 1 liter 0.79 1.27 1.00 1.55 1.13 0.959 2 liter 1.29 0.78 2.00 1.66 1.18 0.918 6 pack 2.29 0.44 2.13 1.77 1.25 0.913 3 liter 1.69 0.59 3.00 1.68 1.19 0.877 12 pack 3.59 0.28 4.26 1.73 1.23 0.909 24 pack 5.19 0.19 8.52 1.58 1.14 0.938
Using the Ideal Point to Select Soda Pop Packages Data Criteria Trade-off Values
Item Price (dollars)
Cost (dollars-1)
Quantity (liters)
L1 norm
L2 norm
L norm
1 can 0.50 2.00 0.35 1.34 1.04 0.986 20 oz 0.60 1.67 0.59 1.44 1.07 0.976 1 liter 0.79 1.27 1.00 1.55 1.13 0.959 2 liter 1.29 0.78 2.00 1.66 1.18 0.918 6 pack 2.29 0.44 2.13 1.77 1.25 0.913 3 liter 1.69 0.59 3.00 1.68 1.19 0.877 12 pack 3.59 0.28 4.26 1.73 1.23 0.909 24 pack 5.19 0.19 8.52 1.58 1.14 0.938
04/10/23 © 2009 Bahill
300
The Search Beam techniqueThe Search Beam technique• Construct a vector between the anti-ideal
point, the nadir (the origin in this example), and the ideal point, then re-examine solutions close to this vector.
• The nadir might be the point where each criterion takes on its minimum value, or it might be the status quo.
• The 6 pack and 3 liter bottle are closest to this vector. Of these, the 3 liter bottle is closest to the ideal point, so it is chosen.
04/10/23 © 2009 Bahill
301
Search BeamSearch Beam22
Use of the Ideal Point
0
1
2
3
0 10 20
Quantity (liters)
Co
st
(1/d
oll
ars
) Ideal Point
The Search Beam
Nadir
04/10/23 © 2009 Bahill
302
Fuzzy Logic, rationaleFuzzy Logic, rationale•Some things are described well by probability theory. Such as the probability that John Wayne was a tall person is around 1.0.
•But what is the probability that George W. Bush is a tall person?
•This question does not have a good answer.
•The theory of Fuzzy Logic was invented to model such questions.
•With fuzzy logic the question becomes, “What is the possibility that George W. Bush belongs to the set of people called tall?”
04/10/23 © 2009 Bahill
303
Fuzzy Logic, exampleFuzzy Logic, example•Here is a fuzzy set for tall people.
•Of course, it could be refined for males or females, old or young people, and for country of origin.
Fuzzy Set for Tall People
Tall
7068
1.0
0.0
De
gre
e o
fM
em
be
rsh
ip
Height (inches)
72
04/10/23 © 2009 Bahill
304
Fuzzy Sets for PerformanceFuzzy Sets for Performance
Five Fuzzy Sets for the Performance Figure of Merit
Very High HighMedium LowVery Low
3210
1.0
0.0
De
gre
e o
fM
em
bers
hip
Quantity (liters)
4
04/10/23 © 2009 Bahill
305
Fuzzy Sets for CostFuzzy Sets for Cost
Very Low LowMedium HighVery High
0.51.01.52.02.5
1.0
0.0
Deg
ree o
fM
em
bers
hip
Cost (1/dollars)
Five Fuzzy Sets for the Cost Figure of Merit
0
04/10/23 © 2009 Bahill
306
Fuzzy rules for a single can Rule number Fuzzy premises Consequences
Cost Volume 1 Very Low Very Low 1 Can 2 Very Low Low 1 Can 3 Very Low Medium 1 Can 4 Very Low High 1 Can 5 Very Low Very High 1 Can 6 Low Very Low 1 Can 7 Low Low 1 Can 8 Low Medium 1 Can 9 Low High 1 Can
10 Low Very High 1 Can 11 Medium Very Low 1 Can 12 Medium Low 1 Can 13 Medium Medium 1 Can 14 Medium High 1 Can 15 Medium Very High 1 Can 16 High Very Low 1 Can 17 High Low 1 Can 18 High Medium 1 Can 19 High High 1 Can 20 High Very High 1 Can 21 Very High Very Low 1 Can 22 Very High Low 1 Can 23 Very High Medium 1 Can 24 Very High High 1 Can 25 Very High Very High 1 Can
04/10/23 © 2009 Bahill
307
Degree of fulfillmentDegree of fulfillment
• Assume premises are connected by ANDs
• Use product rule for AND
04/10/23 © 2009 Bahill
308
Single can, degree of fulfillment (DoF) Rule
number Cost Volume Package DoF
1 Very Low 0.00 Very Low 0.65 1 Can 0.00 2 Very Low 0.00 Low 0.35 1 Can 0.00 3 Very Low 0.00 Medium 0.00 1 Can 0.00 4 Very Low 0.00 High 0.00 1 Can 0.00 5 Very Low 0.00 Very High 0.00 1 Can 0.00 6 Low 0.00 Very Low 0.65 1 Can 0.00 7 Low 0.00 Low 0.35 1 Can 0.00 8 Low 0.00 Medium 0.00 1 Can 0.00 9 Low 0.00 High 0.00 1 Can 0.00 10 Low 0.00 Very High 0.00 1 Can 0.00 11 Medium 0.00 Very Low 0.65 1 Can 0.00 12 Medium 0.00 Low 0.35 1 Can 0.00 13 Medium 0.00 Medium 0.00 1 Can 0.00 14 Medium 0.00 High 0.00 1 Can 0.00 15 Medium 0.00 Very High 0.00 1 Can 0.00 16 High 0.00 Very Low 0.65 1 Can 0.00 17 High 0.00 Low 0.35 1 Can 0.00 18 High 0.00 Medium 0.00 1 Can 0.00 19 High 0.00 High 0.00 1 Can 0.00 20 High 0.00 Very High 0.00 1 Can 0.00 21 Very High 1.00 Very Low 0.65 1 Can 0.65 22 Very High 1.00 Low 0.35 1 Can 0.35 23 Very High 1.00 Medium 0.00 1 Can 0.00 24 Very High 1.00 High 0.00 1 Can 0.00 25 Very High 1.00 Very High 0.00 1 Can 0.00
04/10/23 © 2009 Bahill
309
Rules with non-zero degree of fulfillment (DoF) Rule number
Cost Volume Package DoF
21 Very High 1.00 Very Low 0.65 1 Can 0.65 22 Very High 1.00 Low 0.35 1 Can 0.35 37 Medium 0.46 Low 1.00 1 liter 0.46 42 High 0.54 Low 1.00 1 liter 0.54 58 Low 0.44 Medium 1.00 2 liter 0.44 63 Medium 0.56 Medium 1.00 2 liter 0.56 78 Very Low 0.12 Medium 0.87 6 pack 0.10 79 Very Low 0.12 High 0.13 6 pack 0.02 83 Low 0.88 Medium 0.87 6 pack 0.77 84 Low 0.88 High 0.13 6 pack 0.11 109 Low 0.82 High 1.00 3 liter 0.82 114 Medium 0.18 High 1.00 3 liter 0.18 125 Very Low 0.44 Very High 1.00 12 pack 0.44 130 Low 0.56 Very High 1.00 12 pack 0.56 150 Very Low 0.62 Very High 1.00 24 pack 0.62 155 Low 0.38 Very High 1.00 24 pack 0.38
04/10/23 © 2009 Bahill
310
Can we use this fuzzyCan we use this fuzzyrule base to give advice?rule base to give advice?11• Suppose our customer says, “I
want a little bit of soda pop.”
• We would convert that to, “Cost= don’t care AND Quantity = Very Low.”
• The rule base recommends, “Buy a single can DoF = 0.65.”
04/10/23 © 2009 Bahill
311
Can we use this fuzzyCan we use this fuzzyrule base to give advice?rule base to give advice?22
• Suppose our customer says, “A few of my friends and I cashed in all our empty bottles. We want to buy some soda pop and put it in this little cooler.”
• We would convert that to, “Cost = Low AND Quantity = Medium.”
• Two rules succeed: one for the 2 liter bottle and one for the 6 pack. The highest DoF is for the 6 pack. Therefore, we would recommend, “Buy a 6 pack, DoF = 0.77.”
04/10/23 © 2009 Bahill
312
Can we use this fuzzyCan we use this fuzzyrule base to give advice?rule base to give advice?33• Suppose our customer has a picnic cooler full
of ice and says, “I want a lot of soda pop.”
• We would convert that to, “Cost = don’t care AND Quantity = Very High.”
• Two rules succeed for the 12 pack. Using a sum minus product combining rule, we would recommend, “Buy a 12 pack, DoF = 0.75.”
• However, two rules also succeed for the 24 pack. Using the same combining rule, we would also recommend, “Buy a 24 pack, DoF = 0.76.”
04/10/23 © 2009 Bahill
313
The technique used determines the resultThe technique used determines the resultTechnique Preferred
alternative------------------------------------------------------------------
---Sum 24 packProduct 3 liter bottleSum Minus Product 24 packCompromise 24 packIdeal point
L1 norm single canL2 norm single canL infinity 3 liter bottleModified Minkowski 12 pack
Search beam 3 liter bottleFuzzy rule base 6, 12 or 24 pack
04/10/23 © 2009 Bahill
314
Technique used determines the resultTechnique used determines the result22
But by clever selection of weights and scoring functions we could also get the 20 ounce, the one liter and two liter bottles.
04/10/23 © 2009 Bahill
315
Decision treesDecision trees**
•Another, not necessarily tradeoff study, tool for decision analysis and resolution.
•Example key decisions and their alternatives Is formal evaluation needed? [yes, no] Evaluation data source? [approximations, analysis,
models and simulations, experiments, prototypes] Combining function? [sum, product, sum minus
product, compromise] Alternatives? [alt-1, alt-2, alt-3] Question order may be important, e. g. ask about
dog system function before fertility.
OK, the next slide is the decision tree for these questions.
04/10/23 © 2009 Bahill
316
Decision must be made
yes
prototypes
experiments
models andsimulations
analysis
approximation
compromise
sum minus product
product
sum
3
compromise
sum minus product
product
sum
3
compromise
sum minus product
product
sum
3
compromise
sum minus product
product
sum
3
compromise
sum minus product
product
sum
3
2
Input data source?
Combining function?
Is formal evaluation needed?
no
1
alt-1
alt-2
alt-3
4
alt-1
alt-2
alt-3
4
Alternative?
4
4
4
alt-1
alt-2
alt-3
4
alt-1
alt-2
alt-3
4
alt-1
alt-2
alt-3
4
1
2
3
5
4
7
6
10
9
8
12
11
60
59
58
04/10/23 © 2009 Bahill
317
Killer tradesKiller trades•We do not have time to analyze all 60 possibilities. So we limit the number of things to be studied by doing killer trades. That is, we answer certain questions and kill off large parts of the decision tree.
•In this example we will say that a formal evaluation is necessary, we will use approximation data and the sum combining function.
•This means that our tradeoff study matrix only needs three columns, one for each alternative.
04/10/23 © 2009 Bahill
318
Tradeoff study by a baseball manager
Alternatives → Criteria ↓
Present pitcher
Right-hand short reliever
Left-hand short reliever
Right-hand long reliever
Left-hand long reliever
Pitcher effectiveness
Inning Men on base Score Bullpen readiness
04/10/23 © 2009 Bahill
319
Who should be pitching for us?
yes
Should I use a long or short
reliever?Should I pull the present pitcher?
no
1
Short reliever
Long reliever
2
LHP
RHP
3
LHP
RHP
3
Right-hand or left-hand
pitcher?
Decision Tree for a Baseball Manager
04/10/23 © 2009 Bahill
320
Should we walk this famous slugger?Should we walk this famous slugger?
*Includes hit by pitch, error, etc.**Indicates preferred option†Utility is runs plus expected future runs, from an initial condition of no runners on base and no outs. For the pitching team, less utility is best.
Options Outcomes (Probability) Utility†
Pitchto him
Intentional walk
Homerun
Triple
Double
Single
Walk*
Out
Walk
Walk or
Pitch?
(1.0)
0.67**
0.9
(0.09)
(0.01)
(0.04)
(0.15)
(0.20)
(0.53)
1.5
1.4
1.2
0.9
0.9
0.3
0.9
∑
04/10/23 © 2009 Bahill
321
Some Cautions from Decision TheorySome Cautions from Decision Theory
04/10/23 © 2009 Bahill
322
ValuesValues•Your job is to help a decision maker make
valid decisions.
•This is a difficult and iterative task.
• It entails discovering the decision makers weights of importance, scoring functions, and preferred combining functions.
•You must get into the head of the decision maker and discover his or her preferences and values*
04/10/23 © 2009 Bahill
323
Personality typesPersonality types•Different people have different personality types.
•The Myers-Briggs model is one way of describing these personality types.
•Sensory - Thinking – Judging people are likely to appreciate the tradeoff study techniques we have presented.
•Intuitive – Feeling people most likely will not.
04/10/23 © 2009 Bahill
324
PhrasingPhrasing•The way you phrase the question may determine the answer you will get.
•When asked whether they would approve surgery in a hypothetical medical emergency, many more people accepted surgery when the chance of survival was given as 99 percent than when the chance of death was given as 1 percent.
04/10/23 © 2009 Bahill
325
Preference ReversalsPreference Reversals**
$ betHas higher dollar
value
P betHas higher probability
Although the expected values are the same,
most people preferred to play the P bet, however
most people wanted a higher selling price for the $ bet.
Lichtenstein & Slovic (1971)
$5.40
$0$56.70
$0
04/10/23 © 2009 Bahill
326
Factors affecting human decisionsFactors affecting human decisions the decision maker corporate culturethe decision maker’s valuespersonality typesrisk aversenessbiases, illusions and use of heuristics
information displayedwording of the questioncontext
the decisioneffort required to make the decisiondifficulty of making the decisiontime allowed to make the decisionneeded accuracy of the decisioncost of the decisionlikelihood of regret
04/10/23 © 2009 Bahill
327
Temporal orderTemporal order•You will get more consistent results if youfirst work on the criteria then fill in the matrix of evaluation data row by rowassign weights last, that way criteria that have no affect on the outcome can be given minimal weights
04/10/23 © 2009 Bahill
328
When you get When you get “The Wrong Answer” “The Wrong Answer” you could change you could change
• Weights of importance• Scores for the alternatives• Parameters of the scoring functions• Parameters of the combining function• The combining function itself• The tradeoff method
04/10/23 © 2009 Bahill
329
But we think,But we think,If you got the wrong answer,
then you got the requirements wrong.
04/10/23 © 2009 Bahill
330
04/10/23 © 2009 Bahill
331
Possible missing requirementsPossible missing requirements• Need for Storage Space• Time Before Soda Loses Carbonization• Need for a Glass• Availability of Cold Soda in the Desired Size• Ziggy’s Trips to the Restroom
04/10/23 © 2009 Bahill
332
The feeling in your stomach testThe feeling in your stomach test**
•Assume you are trying to make an important decision, like “Should I quit my job and become a consultant?”
•You have done a tradeoff study, but the results are equivocal.
•How should you decide?
•Get a coin. Assign heads and tails, e.g. heads I quit my job, tails I keep my job. Flip the coin and look at the result. What is the immediate feeling in your stomach?
•If it was heads, but your stomach is in turmoil, then keep your job.
04/10/23 © 2009 Bahill
333
LimitationsLimitations
04/10/23 © 2009 Bahill
334
LimitationsLimitations•Limited time and resources guarantee that a tradeoff study will never contain all possible criteria.
•Tradeoff studies produce satisficing (not optimal) solutions.
•A tradeoff study reflects only one view of the problem. Different tradeoff analysts might choose different criteria and weights and therefore would paint a different picture of the problem.
•We ignored human decision-making mistakes for which we have no corrective action, such as closed mindedness, lies, conflict of interest, political correctness and favoritism.
04/10/23 © 2009 Bahill
335
UncertaintyUncertainty•We studied two independent tradeoff studies that had a variability or uncertainty statistic associated with each evaluation datum.
•These statistics were carried throughout the whole computational process, so that at the end the recommended alternatives had associated uncertainty statistics.
•Both of these studies were incomprehensible.
•Therefore, we did not try to accommodate uncertainty, changes and dependencies in the evaluation data.
04/10/23 © 2009 Bahill
336
Speed BumpSpeed Bump
04/10/23 © 2009 Bahill
337
A Tradeoff Study of A Tradeoff Study of Tradeoff Study ToolsTradeoff Study Tools
04/10/23 © 2009 Bahill
338
COTS-Based Engineering ProcessCOTS-Based Engineering Process•When choosing commercial off the shelf
(COTS) products the following generic criteria may be convenient: Percent of requirements satisfied Vendor viability Total life cycle cost Apparent interface ease Architectural compatibility Foreign components User interface ease of use Observable states
04/10/23 © 2009 Bahill
339
Specific criteriaSpecific criteria• For tradeoff study tools these specific criteria may be
convenient: Rationale is easy to understand Can verify calculations with paper and pencil Works with nonconvex distributions of alternatives Implements scoring functions (utility curves) Has multiple combining functions Performs sensitivity analyses
04/10/23 © 2009 Bahill
340
A tradeoff study on A tradeoff study on tradeoff study toolstradeoff study tools•A tradeoff study was performed starting with 60 COTS decision analysis tools.
•These were the final Preferred alternatives Pinewood by Bahill Intelligent Computer Systems Hiview by Catalyze Ltd. Logical Decisions for Windows by Logical Decisions
Inc. Expert Choice by Expert Choice Inc.
See A Tradeoff Study of Tradeoff Study Tools http://www.sie.arizona.edu/sysengr/sie554/tradeoffStudyOfTradeoffStudyTools.doc
04/10/23 © 2009 Bahill
341
Use Cases fromUse Cases fromA Tradeoff Study of Tradeoff Study ToolsA Tradeoff Study of Tradeoff Study Tools
04/10/23 © 2009 Bahill
342
Architecture of a tradeoff study toolArchitecture of a tradeoff study tool
Alt-3Criteria-1Criteria-2Criteria-3 SubCrit-3.1 SubCrit-3.2 SubCrit-3.3Criteria-4
Alt-2Criteria-1Criteria-2Criteria-3 SubCrit-3.1 SubCrit-3.2 SubCrit-3.3Criteria-4
Alt-1 Alt-2 Alt-3 Alt-4 Alt-5Criteria-1Criteria-2Criteria-3 SubCrit-3.1 SubCrit-3.2 SubCrit-3.3Criteria-4
Alt-1 Alt-2 Alt-3 Alt-4 Alt-5Criteria-1Criteria-2Criteria-3Criteria-4Overall Score
Alt-1Criteria-1Criteria-2Criteria-3 SubCrit-3.1 SubCrit-3.2 SubCrit-3.3Criteria-4
Input Module
Output Matrices Summary Module
Criteria Module
Limits, slopes, baselines and weights
04/10/23 © 2009 Bahill
343
Use case diagramUse case diagramud TradeoffStudyTool
Tradeoff Study Tool
Tradeoff Analyst
Create a Tradeoff Study
Complete Criteria Module
Fill In Input Module
Company Resources
PAL
«include»
«include»
04/10/23 © 2009 Bahill
344
Create a Tradeoff StudyCreate a Tradeoff Study**11
Iteration: 2.1Brief Description: Tradeoff Analyst completes the four modules of the tradeoff study tool and gives the results to the decision maker. Every aspect of a tradeoff study requires extensive discussion with the decision maker and other stakeholders.
Added Value: This helps a decision maker to make better decisions and it documents the process that was used to make these decisions.
Level: User goalScope: Applies to a decision problem that is appropriate for a tradeoff study.
Primary Actor: Tradeoff Analyst (this could be a person or a team).
04/10/23 © 2009 Bahill
345
Create a Tradeoff StudyCreate a Tradeoff Study22
Supporting Actors: Tradeoff Analyst will get the tradeoff study tool and documents from Company Resources. Tradeoff Analyst will put the results of the tradeoff study in the project assets library (PAL).
Frequency: Company wide, once a week
Precondition: A decision maker has asked Tradeoff Analyst to perform a tradeoff study. Preliminary criteria, weights, alternatives and criteria values must already be defined and be in the hands of Tradeoff Analyst.
Trigger: Tradeoff Analyst starts the process.
04/10/23 © 2009 Bahill
346
Create a Tradeoff StudyCreate a Tradeoff Study33
Main Success Scenario:1. Tradeoff Analyst copies the company tradeoff study spreadsheet into his or her computer.
2. Tradeoff Analyst selects the Criteria Module for development.
3. Include Complete Criteria Module.4. Tradeoff Analyst selects the Input Module for development.
5. Include Fill Input Module.6. The system transfers data from the Criteria Module into the Output Matrices.
7. The system computes preferred alternatives using the combining function chosen by Tradeoff Analyst.
04/10/23 © 2009 Bahill
347
Create a Tradeoff StudyCreate a Tradeoff Study44
Main Success Scenario (continued):8. The system transfers data from the Output Matrices into the Summary Module.
9. The system displays the Summary Module for Tradeoff Analyst’s inspection.
10. Tradeoff Analyst looks at the preferred alternatives in the Summary Module.
11. Tradeoff Analyst repeats steps 2 to 10 until he or she is satisfied.
12. Tradeoff Analyst submits the tradeoff study for expert review.
13. Tradeoff Analyst submits the tradeoff study to the decision maker and places it in the Process Asset Library (PAL) [exit use case]
04/10/23 © 2009 Bahill
348
Create a Tradeoff StudyCreate a Tradeoff Study55
Unanchored Alternate Flow:
Tradeoff Analyst can stop the system at any time; all entered data and intermediate results will be saved [exit use case].
Postcondition: Tradeoff Analyst has planed a tradeoff study.
Specific Requirements
Functional Requirements:
Note: Transferring data from the Criteria Module into other modules and interchanging information with Company Resources and the PAL are supplementary requirements.
04/10/23 © 2009 Bahill
349
Create a Tradeoff StudyCreate a Tradeoff Study66
Functional Requirements (continued):FR1-1 The system shall compute preferred alternatives using the combining function chosen by Tradeoff Analyst.
FR1-2 The system shall transfer information from the Output Matrices into the Summary Module.
FR1-3 The system shall display the Summary Module.
Nonfunctional Requirements:NFR1 At least six different combining functions shall be available for use by Tradeoff Analyst.
Author/owner: Terry BahillLast changed: February 23, 2006
04/10/23 © 2009 Bahill
350
Concrete inclusion use casesConcrete inclusion use casesThe next two use cases are concrete inclusion use cases to the Create a Tradeoff Study use case.
04/10/23 © 2009 Bahill
351
Complete Criteria ModuleComplete Criteria Module11
Iteration: 2.1Brief Description: Tradeoff Analyst enters data into the Criteria Module and designs scoring functions. If this inclusion use case is called by the base use case, then it is context sensitive; the spreadsheet that is open is the spreadsheet that is used. If the actor initiates the use case, then the name of the spreadsheet to be used must be queried.
Added Value: Tradeoff Analyst understands the criteria and develops scoring functions.
Level: Low levelScope: Criteria ModulePrimary Actor: Tradeoff Analyst
04/10/23 © 2009 Bahill
352
Complete Criteria ModuleComplete Criteria Module22
Frequency: Company wide, once a weekPrecondition: Criteria must already be defined and be in the hands of Tradeoff Analyst.
Trigger: This use case is initiated by the Create a Tradeoff Study use case or by the Tradeoff Analyst.
Main Success Scenario:1a. When triggered by the Create a Tradeoff Study use case, Tradeoff Analyst replaces criteria of the template with problem domain criteria and describes these criteria in the notes section.
2. Tradeoff Analyst works on the criteria one at a time and may rewrite, decompose or derive criteria.
04/10/23 © 2009 Bahill
353
Complete Criteria ModuleComplete Criteria Module33
Main Success Scenario (continued):3. Tradeoff Analyst selects limits, slopes and baselines for the scoring function of each criterion.
4. The system draws a scoring function for each criterion.
5. Tradeoff Analyst readjusts limits, slopes and baselines for each criterion. This requires discussion with the decision maker.
6. The system redraws the scoring function for each criterion.
7. Tradeoff Analyst assigns a weight of importance to each criterion.
8. The system computes normalized weights.
04/10/23 © 2009 Bahill
354
Complete Criteria ModuleComplete Criteria Module44
Main Success Scenario (continued):9. The system displays alternative combining functions and accepts the function chosen by Tradeoff Analyst.
10. Tradeoff Analyst repeats this process until satisfied with the results.
11. Tradeoff Analyst expresses desire to finish this use case.
12. The system transfers criteria to the Input Module [exit use case].
Anchored Alternate Flow:1b. When triggered by the Tradeoff Analyst, Tradeoff Analyst specifies the file to be worked on.
04/10/23 © 2009 Bahill
355
Complete Criteria ModuleComplete Criteria Module55
Unanchored Alternate Flow:Tradeoff Analyst can stop the system at any time; all entered data and intermediate results will be saved [exit use case].
Postcondition: Tradeoff Analyst knows what the criteria are and where they are stored.
Specific RequirementsFunctional Requirements:FR2-1 The Criteria Module shall accept scoring function parameters from Tradeoff Analyst.
FR2-2 The Criteria Module shall create and graph scoring functions.
FR2-3 The Criteria Module shall accept changes in scoring function parameters and criteria from Tradeoff Analyst.
04/10/23 © 2009 Bahill
356
Complete Criteria ModuleComplete Criteria Module66
Functional Requirements (continued):FR2-4 The Criteria Module shall accept un-normalized weights from Tradeoff Analyst.
FR2-5 The Criteria Module shall normalize the weights.
FR2-6 The Criteria Module shall accept changes in weights from Tradeoff Analyst.
FR2-7 The Criteria Module shall display alternative combining functions and accept the function chosen by Tradeoff Analyst.
04/10/23 © 2009 Bahill
357
Complete Criteria ModuleComplete Criteria Module77
Nonfunctional Requirements:NFR2-1 Scoring function graphs must be updated within 100 milliseconds of a change in a parameter.
NFR2-2 Computing normalized weights shall take less than 100 milliseconds.
Business Rules:BR-1. The weights entered by Tradeoff Analyst shall be numbers (usually integers) in the range of 0 to 10, where 10 is the most important.
04/10/23 © 2009 Bahill
358
Fill Input ModuleFill Input Module11
Iteration: 2.1Brief Description: Tradeoff Analyst enters criteria values for the alternatives into the Input Module. If this inclusion use case is called by the base use case, then it is context sensitive, the spreadsheet that is open is the spreadsheet that is used. If the actor initiates the use case, then the name of the spreadsheet to be used must be queried.
Added Value: These criteria values can be used to compute preferred alternatives.
Level: Low levelScope: Input ModulePrimary Actor: Tradeoff AnalystFrequency: Company wide, once a week
04/10/23 © 2009 Bahill
359
Fill Input ModuleFill Input Module22
Precondition: Alternatives must already be defined and their preliminary criteria values must be in the hands of Tradeoff Analyst.
Trigger: This use case is triggered by the Create a Tradeoff Study use case or by the Tradeoff Analyst.
Main Success Scenario:1a. When triggered by the Create a Tradeoff Study use case, Tradeoff Analyst describes his or her alternatives.
2. The system updates the Input Module.3. Tradeoff Analyst concentrates on one row at a time and fills in criteria values for the alternatives.
04/10/23 © 2009 Bahill
360
Fill Input ModuleFill Input Module33
Main Success Scenario (continued):4. Tradeoff Analyst reassesses the criteria values until satisfied with the results.
5. The Input Module sends criteria values to the Criteria Module [exit use case].
Anchored Alternate Flow:1b. When triggered by the Tradeoff Analyst, Tradeoff Analyst specifies the file to be worked on.
Unanchored Alternate Flow:Tradeoff Analyst can stop the system at any time; all entered data and intermediate results will be saved [exit use case].
04/10/23 © 2009 Bahill
361
Fill Input ModuleFill Input Module44
Postcondition: Tradeoff Analyst knows where the alternatives are described and where their criteria values are stored.
Specific RequirementsFunctional Requirements: FR3-1 The Input Module shall accept criteria values from Tradeoff Analyst.
FR3-2 The Input Module shall accept changes in criteria values from Tradeoff Analyst.
Author/owner: Terry BahillLast changed: February 25, 2006
04/10/23 © 2009 Bahill
362
Supplementary requirementsSupplementary requirements•SR1 The system shall interchange information with Company Resources and the PAL.
•SR2 The Criteria Module shall transfer information to and from the Input Module.
•SR3 The Criteria Module shall transfer information to and from the Output Matrices.
SummarySummary
04/10/23 © 2009 Bahill
364
SummarySummary11
•Decompose criteria into subcriteria •Put subcriteria in separate columns •Normalize weights •Derive evaluation data approximations product literature analysis models and simulations experiments prototypes
•Create scoring functions •Combine data in separate areas•Add columns for alternatives
including Do Nothing
04/10/23 © 2009 Bahill
365
SummarySummary22
• There are many multicriterion decision making techniques
• Often they give different recommendations
• If the alternatives form a nonconvex set, then many techniques will have difficulty
• If you got the “wrong answer,”“wrong answer,” then you probably got the requirements wrong
04/10/23 © 2009 Bahill
366
SummarySummary33
• You should use a formal, mathematical technique to evaluate alternative designs
• Standards (e.g. CMMI) require it
• Government organizations require it
• Company policy requires it
• Common sense requires it
• But when you do, be careful or mere artifacts will determine your recommendation
04/10/23 © 2009 Bahill
367
SummarySummary44
• Good industry practices for ensuring success of tradeoff studies include having teams evaluate the data evaluating the data with many iterations peer review of the results and
recommendations
04/10/23 © 2009 Bahill
368
SpeculationSpeculationObservation
As you do a better job of getting the requirements right, the preferred alternatives of different teams converge.
Speculation
As you do a better job of getting the necessary and sufficient requirements, the preferred alternatives of the various tradeoff combining techniques will converge.
04/10/23 © 2009 Bahill
369
SummarySummary55
•Getting an answer is not the most important facet of a tradeoff study.
•Documenting the tradeoff process and the data is often the most important contribution. Think about the San Diego County airport site
selection
•Corporate culture and the decision maker’s personality determine how well the recommendations of a tradeoff study will be received.
•Doing a tradeoff study will help you get the requirements right.
04/10/23 © 2009 Bahill
370
SummarySummary66
•Emotions, illusions, biases and use of heuristics make humans far from ideal decision makers.
•Using tradeoff studies thoughtfully can help move your decisions from the normal human decision-making lower-right quadrant to the ideal decision-making upper-left quadrant.
Dog System
Exercise
04/10/23 © 2009 Bahill
372
Tradeoff study exercise, generalTradeoff study exercise, general1. Find the folder named SandiaDogSelector on
the desktop of your computer.
2a. Open it, read dogProb0.doc and do the exercise.
2b. Wait for the instructor
2c. Read dogSol0.doc
3. Read dogProb1.doc and do the exercise
4. Wait for the instructor
5. Read dogSol1.doc
6. Wait for the instructor
7. Read dogProb2.doc and do the exercise
8. Wait for the instructor
Etc.
04/10/23 © 2009 Bahill
373
Tradeoff study exercise, detailsTradeoff study exercise, details0. Read the problem statement (dogProb0.doc) and write
some preliminary requirements, 5 minutes, wait for solutions (dogSol0.doc), 2 minute discussion.
1. Identify key system decisions and their alternatives (dogProb1.doc). 8 minutes, wait for solutions (dogSol1.doc), 7 minute discussion.
2. Fill in the Decision Tree Worksheet, use text boxes or do it on paper (dogProb2.doc). 8 minutes, wait for solutions (dogSol2.doc), 7 minute discussion.
3. Use the Decision Resolution Worksheet (dogProb3.doc) to perform the Killer Trades. 8 minutes, wait for solutions (dogSol3.doc).
4. Define the tradeoff studies that still need to be done and list them on the Decision Resolution Worksheet (dogProb4.doc). 5 minutes, wait for solutions (dogSol4.doc).
04/10/23 © 2009 Bahill
374
Tradeoff study exercise, detailsTradeoff study exercise, details5. List evaluation criteria and weights of
importance on the Criteria Description Worksheet (dogProb5.doc). 20 minutes, wait for solutions (dogSol5.doc), 5 minute discussion.
04/10/23 © 2009 Bahill
375
You can get criteria from your
PAL
04/10/23 © 2009 Bahill
376
Tradeoff study exerciseTradeoff study exercise6. Perform a tradeoff study using the Tradeoff
Matrix Spreadsheet (dogProb6.xls). 30 minutes, wait for solutions, In 10 minutes discuss both dogSol6.xls and dogSol6.doc.
For scoring functions open the folder named SSF and use the tool named SSF.exe
7. Fix the Do Nothing problem (dogProb7.doc and dogProb7.xls). 5 minutes, wait for solutions. In 5 minutes discuss the sensitivity analysis in dogSol7.doc and dogSol7.xls.
04/10/23 © 2009 Bahill
377
Tradeoff study exerciseTradeoff study exercise8. Recompute your tradeoff matrix using a
combining function other than the sum of weighted scores (dogProb8.doc and dogProb8.xls). 15 minutes, wait for solutions. In 5 minutes discuss the sensitivity analysis in dogSol8.doc and also the solutions in dogSol8.xls.
Mathematical Summary of Tradeoff Techniques
04/10/23 © 2009 Bahill
379
EquationsEquations•The following section uses algebraic equations to summarize the tradeoff methods we have just discussed. These slides are located at
www.sie.arizona.edu/sysengr/slides/tradeoffMath.doc
•If you are equation intolerant, you can leave now and we won’t be offended.
•Or, if in the middle of the presentation you find that you have exceeded your equation viewing limit, you may leave.
•Please fill out a course evaluation questionnaire before you go.
04/10/23 © 2009 Bahill
380
04/10/23 © 2009 Bahill
381
Acronym Acronym listlist
AHP Analytic Hierarchy Process BCS Bowl Championship Series BM Basic Measure CDR Critical Design Review CF Combining Function CMMI Capability Maturity Model Integrated COTS Commercial Off The Shelf DAR Decision Analysis and Resolution DM Decision Maker DoF Degree of Fulfillment EV Expected Value IPT Integrated Product Development Team IQ Intelligence Quotient MAUT Multi-Attribute Utility Technique NFL National Football League NOP Normal Operating Point PAL Process Asset Library PC Personal Computer PDR Preliminary Design Review QFD Quality Function Deployment SEMP Systems Engineering Management Plan SRR System Requirements Review Wt Weight
04/10/23 © 2009 Bahill
382
Course materialsCourse materials•This slide show, we present this in Vista For the “Humans are not rational2” slide, bring two $2
bills, a coin, two $1 bills, a lottery ticket and the last two slides of this presentation.
•Dog System Exercise problems and solutions this is 21 files plus one folder we need computers for this exercise Load the files onto the desktop of the PCs before the
class
•Mathematical Summary MS Word Slides
•The student computers will need PowerPoint, MS Word and Excel.
•Optional handouts include Ben Franklin’s letter and the GOAL/QPC Creativity Tools Memory Jogger.
04/10/23 © 2009 Bahill
383
HistoryHistory•This course is based on material from Terry
Bahill’s Systems Engineering Process course at the University of Arizona.
•Bahill adapted it for BAE in the Fall of 2004 where it was reviewed by Rob Culver, Bill Wuersch, and John Volanski and it was piloted October 12-13, 2004.
•The human decision making material was added at the UofA in Fall 2005.