using analytical hierarchy process to select a job

EMSE 388 – Quantitative Methods in Cost Engineering

Lecture Notes by Instructor: Dr. J. Rene van Dorp Chapter 20 - Page 229 Source: Financial Models Using Simulation and Optimization by Wayne Winston

USING ANALYTICAL HIERARCHY PROCESS TO SELECT A JOB

When multiple objectives are important to a Decision Maker,

it is often difficult to choose between alternatives.

Example: You are choosing a job. One might offer the highest starting salary, but rate

poorly on your other objectives: quality of life, closeness to your family. Another

job offer might rate highly on these latter objectives but has a relatively low

starting salary. Which one do you choose?

THOMAS SAATY’s ANALYTICAL HIERARCHY PROCESS Provides a powerful tool that can be used to make decisions

when multiple objectives are present.


Lecture Notes by Instructor: Dr. J. Rene van Dorp Chapter 20 - 30 Source: Financial Models Using Simulation and Optimization by Wayne Winston

Four Objectives: Objective 1: High Starting Salary

Objective 2: Quality of life in city where job is located

Objective 3: Interest of Work

Objective 4: Nearness of job to family.

• The AHP process generates a weight for each objective. By convention the

weights sum up to 1.

Weights For Objectives:

OBJECTIVE DESCRIPTION Weight Objective 1 Salary 0.5115 Objective 2 Life quality 0.0986 Objective 3 Work interest 0.2433 Objective 4 Near family 0.1466

TOTAL WEIGHT 1.0000



CONCLUSION:

• High Starting Salary (HSS) is your most important objective.

• HSS is twice as important as the interest of the work.

• HSS is approximately three times as important as nearness to your family.

• HSS is approximately five times as important Quality of Lie in the city where

you work.

YOU HAVE OBTAINED THREE JOB OFFERS The following Table described how each Job scores on the four objectives.

Determining best job Matrix of scores

Salary Life quality Work interest Near familyJob 1 0.5714 0.1593 0.0882 0.0824 Job 2 0.2857 0.2519 0.6687 0.3151 Job 3 0.1429 0.5889 0.2431 0.6025

TOTAL WEIGHT1.0000 1.0000 1.0000 1.0000



CONCLUSION:

• Job 1 is twice as important in terms of salary as job 2 and three times as

important as job 3.

• Job 3 is twice as important in terms of Quality of Life as job 2 and three

times as important as job 1.

• Job 2 is more than twice as important in terms of Interesting Work as job 3

and eight times as important as job 1.

WHICH ONE DO YOU CHOOSE?



CALCULATE COMBINED SCORES:

Job 1 Score = 0.5115*0.5714+0.0986*0.1593+

0.2433*0.0882+0.1466*0.0824 = 0.3415

Job 2 Score = 0.5115*0.2857+0.0986*0.2519+

0.2433*0.6687+0.1466*0.3151 = 0.3799

Job 3 Score = 0.5115*0.1429+0.0986*0.5889+

0.2431*0.6687+0.1466*0.6025 = 0.2786

Conclusion: Job 2 gets the highest score. Hence, the analysis above recommends taking job

number 2.



Note that:

• Combined Score for Jobs is a Matrix-Vector product that can be easily

calculated using the MMULT function in Excel.

Inputs to the above analysis are:

• Weights for each objective

• Scores for each Alternative on each of the Objectives

OUTPUT OF ANALYTICAL HIERARCHY PROCESS PROVIDES THESE WEIGHTS AND SCORES!!



PAIRWISE COMPARISON MATRIX OF OBJECTIVES Suppose we have N objectives. Then consider the N*N Matrix A, such that:

=

NNN

N

aa

aaA

,1,

,11,1

where

ai,j= “Importance of Objective i compared to Objective j.

BUT HOW DO WE GET SPECIFIC VALUES FOR ai,j?

<

>=

j Objective asImportant less is i Objective1j Objective asImportant as is i Objective1

j Objective thanimportant more is i Objective1

, jia



ANSWER: THROUGH EXPERT JUDGEMENT QUESTIONNAIRES

INTERMEZZO: WHAT IS EXPERT JUDGMENT?

IT IS NOT! A Group of Experts in

a Room deciding on Numbers.



STRUCTURED APPROACH TO CAPTURING AN EXPERTS KNOWLEDGE BASE

AND CONVERT HIS KNOWLEDGE BASEINTO QUANTITATIVE ASSESSMENTS.

EXPERTS

SUBSTANTIVE

NORMATIVE

1. DECOMPOSITION OF EVENT OF INTERESTTO A MEANINGFULL

LEVEL FOR SUBSTANTIVE EXPERT

KNOWLEDGABLE ABOUT THE SUBJECT MATTER

AND EXTENSIVE EXPERIENCE

MODELERS SKILLED IN DECOMPOSITION

AND AGGREGATION OF ASSESSMENTS

EXPERT JUDGEMENT ELICITATION PROCEDURE

2. ELICITATION OF JUDGMENT OF

SUBSTANTIVE EXPERT FACILITATED

BY NORMATIVE EXPERT

3. AGGREGATION OF JUDGEMENTS

BY NORMATIVE EXPERT

ELICITATION PROCESS =MULTIPLE CYCLES

(AT LEAST 2)



EXAMPLE: THE DELPHI METHOD

• Early 1950: Developed by RAND Corporation as spin-off of an Air Force

Research Rroject, "Project Delphi".

• 1963: Wider audience due to 1963 RAND Study "Report on a long-range

Forecasting study".

Probably, best known method to date of

Eliciting and synthesizing expert judgment.

STEP 1: Monitoring Team defines set of issues and selects sets of Respondents who are

experts on the issues in question. Respondents do not know who other

respondents are, and the responses are anonymous. Preliminary questionnaire

is sent for comments, which are then used to establish a definitive questionnaire.



STEP 2: Questionnaire is sent to respondents. Monitoring Team analyses the answers.

STEP 3: The set of responses is sent back together with 25% lower and 25% upper

responses. The respondents are asked if they wish to revise the initial

predictions. Those who answered outside of the above range are asked to give

arguments.

STEP 4: The revised predictions are analyzed by the monitoring team and the outliers for

arguments are summarized. GOTO STEP 2.

TYPICALLY THREE ROUNDS!



EXAMPLE OF DELPHI QUESTIONNAIRE # 1

Questionnaire # 1 This is the first in a series of four questionnaires intended to demonstrate the use of the

Delphi Technique in obtaining reasoned opinions from a group of respondents.

Each of the following six questions is concerned with developments in the United States

with the next few decades.

In addition to giving your answer to each question, you are also being asked to rank the

questions from 1 to 7. Here “1” means that in comparing your ability to answer this question

with what you expect the ability of the other participants to be, you feel that you have the

relatively best chance of coming closer to the truth than most of the others, while a “7”

means that you regard that chance as relative least.

Source: Helmer (1968)



Rank Answer*

1. In your opinion, in what y ear will the median familiy income (in1967 dollars) reach twice its present amount?2. In what y ear will the percentage of electric automobiles along allautomobile in use reach 50 percent?3. In what y ear will the percentage of households that are equippedwith computer consoles tied to a central computer and databankreach 50 percent?4. By what y ear will the per-capita amount of personal cash transactions(in 1967 dollars) be reduced to one-tenth of what it is now?5. In what y ear will power generation be thermonuclear fusion becomecomercially competitive with hydroelectric power?6. By what y ear will it be possible by commercial carriers to get fromNew York's Time Square to San Fransisco's Union Suare in half thetime that is now required to make that trip?7. In what y ear will a man for the first time travel to the Moon, stayfor at least 1 month, and return to earth?* "Never" is alson an acceptable answer

Please also answer the following question, and give your name (this is for identification purposesduring the exercise only ; no opinions will be attributed to a particular person).

Check One: I would like

I am willing but no anxious

I would prefer not

participate in the three remaining qustionnaires

Name (block letter please):………………………………………..



CRITIQUE ON DELPHI METHOD: (Sackman's Delphi Critique (1975))

Methodological

• Questions are vague are often so vague that it would be impossible to

determine when, if ever, they occurred.

• Furthermore, the respondents are not treated equally.

• Many dropouts. No Explanation for # of dropouts given or researched, nor are

effects assessed on eventual assessment. Does Delphi convergence because

of boredom in stead of consensus?

• Sackman argues that experts and non experts produce comparable results.



Comparison to other Methods (Delbecq, Van de Ven, and Gusstafson, 1975);

• Method 1: "nominal group technique"; participants confront each other directly

in a controlled environment.

• Method 2: "no interaction model”; initial assessments are simply aggregated

mathematically.

Results:

• Nominal group technique superior to the others,

• Delphi worst of the three.



EXPERT JUDGMENT ELICITATION PRINCIPLES (Source: Experts in Uncertainty, Roger M. Cooke)

1. Reproducibility: It must be possible for scientific peers to review and if necessary reproduce all

calculations. This entails that the calculation model must be fully specified and

the ingredient data must be made available.

2. Accountability: The source of Expert Judgment must be identified.

3. Empirical Control: Expert probability assessment must in principle be susceptible to empirical

control.



4. Neutrality: The method for combining/evaluating expert judgements should encourage

experts to state true opinions.

5. Fairness: All Experts are treated equally, prior to processing the results of observations

PRACTICAL EXPERT JUDGMENT

ELICITATION GUIDELINES

1. The questions must be clear 2. Prepare an attractive format for the questions and graphic format for the

answers



3. Perform a dry run 4. An Analyst must be present during the elicitation 5. Prepare a brief explanation of the elicitation format, and of the model for

processing the responses. 6. Avoid Coaching 7. The elicitation session should not exceed 1 hour.



ELICITATION PROCEDURES

• Direct Procedures: Ask for Probabilities\Measures of Central

Tendency\Measures of Variability

• Indirect Procedures: Use Betting Strategies; Paired Comparisons

Example Betting Strategies: Indifference when Expected payoffs are the same

1. Betting Stategies

Event: Lakers winning the NBA title this season



STEP 1: Offer a person to choose between following the following bets, where

X=100, Y=0.

Bet for Lakers

Lakers Win

Bet against Lakers

Lakers Win

Lakers Loose

Lakers Loose

Max Profit

X

-X

-Y

Y


X=0, Y=100. (Consistency Check)




X=100, Y=50.


X=50, Y=100. (Consistency Check)

Continue until point of indifference has been reached.

Assumption:

When a person is indifferent between bets the

expected payoffs from the bets must be the same.



Thus:

X*Pr(LW) – Y*Pr(LL)= -X*Pr(LW) + Y*Pr(LL) ⇔

2*X*Pr(LW) – 2*Y*(1- Pr(LW) )=0 ⇔

Pr(LW) = YXY+ .

Example: X=100, Y=50 ⇒ Pr(LW)= 13 ≈ 33.33%



2. Paired Comparisons of Situations

Issaquah class ferry on the Bremerton to Seattle route in acrossing situation within 15 minutes, no other vessels around,

good visibility, negligible wind.

Other vessel is a navy vessel Other vessel is a product tanker



Question: 1 89

Situation 1 Attribute Situation 2 Issaquah Ferry Class -

SEA-BRE(A) Ferry Route - Navy 1st Interacting Vessel Product Tanker

Crossing Traffic Scenario 1st Vessel - 0.5 – 5 miles Traffic Proximity 1st Vessel -

No Vessel 2nd Interacting Vessel - No Vessel Traffic Scenario 2nd Vessel - No Vessel Traffic Proximity 2nd Vessel - > 0.5 Miles Visibility - Along Ferry Wind Direction -

0 Wind Speed - Likelihood of Collision Avoidance 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9

Situation 1 is worse <====================X====================> Situation 2 is worse

9: VERY MUCH MORE LIKELY to result in a collision. 7: MUCH MORE LIKELY to result in a collision. 5: MODERATELY LIKELY to result in a collision. 3: SOMEWHAT MORE LIKELY to result in a collision. 1: EQUALY LIKELY to result in a collision.


Lecture Notes by Instructor: Dr. J. Rene van Dorp Chapter 20 - 3 Source: Financial Models Using Simulation and Optimization by Wayne Winston

Underlying Model for Paired Comparison Questionnaire

)(0

1 1

)X Failure,Propulsion|AccidentPr( XYBTeP=

1X 1 Scenario Traffic = 2X 2 Scenario Traffic =

nsinteractioway -2 includingVector )X(Y =

Paired Comparison

( ) )()(

)2(0

)(0

2

121

1

)X Failure,Prop.|AccidentPr()X Failure,. Prop|AccidentPr( XYXY

XY

XYT

T

T

eePeP −== β

β

β

( ) )()( )X Failure,Prop.|AccidentPr()X Failure,. Prop|AccidentPr( 21

2

1

XYXYLN T −=

β

1

2

3

4

Calibration to Accident Data

Calibrate to convert relative probabilities to absolute probabilities by solving for

P0 e.g. fix the total number of expected collisions over a given time period.



BACK TO THE ANALYTICAL HIERARCHY PROCESS

PAIRWISE COMPARISON MATRIX OF OBJECTIVES TO OBTAIN WEIGHTS FOR OBJECTIVES

Suppose we have N objectives. Then consider the N*N Matrix A, such that:

=

NNN

N

aa

aaA

,1,

,11,1

where

ai,j= “How much more important is objective

i compared to Objective j.



<

>=

j Objective asImportant less is i Objective1j Objective asImportant as is i Objective1

j Objective thanimportant more is i Objective1

, jia

ASSUMPTION:

• There exists a set of numbers iw , i=1,…, n such that:

iw = Relative importance of objective i compared to the other objectives.

j

ijii

n

ii w

waww =>=∑=

,1

,0,1



Note that:

ji

j

ii

jij a

www

wa

,,

11===

BUT HOW DO WE GET SPECIFIC VALUES FOR ai,j? STEP 1: Introduce a quantitative scale for measuring importance

1: Equally Important

3: Slightly More Important

5: Strongly More Important

7: Very Strongly More Important

9: Absolutely More Important



CRITIQUE: Where does this scale come from? Why five categories and why a

scale from 1 to 9? Do the analysis result depend one these choices?

ANSWER: There is instability in the final scores that are being calculated and in

the RANKINGS of the alternatives as well (AHP is not perfect, but very practical

and widely applied method).

STEP 2: Develop a questionnaire. Use an attractive graphical format for the questions.



For example:

Objective ObjectiveQuality of Life Starting Salary

LHS is Preferred RHS is Preferred

9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9

Abs

olut

ely

Mor

e Im

porta

nt

Ver

y S

trong

ly M

ore

Impo

rtan

Stro

ngly

Mor

e Im

porta

nt

Slig

htly

Mor

e Im

porta

nt

Equ

ally

Impo

rtant

Slig

htly

Mor

e Im

porta

nt

Stro

ngly

Mor

e Im

porta

nt

Ver

y S

trong

ly M

ore

Impo

rtan

Abs

olut

ely

Mor

e Im

porta

nt



Total number of objectives is N.

What is the number of pairwise comparison you need to ask?

ANSWER:

2N

Answers are typically summarized in a Matrix Form

=

1212

41

21221

21

211

51

4251

A



Back to our Job Selection Example: HOW DO WE CALCULATE THE WEIGHT VECTOR

Tnwwww ),,,( 21=

FROM THIS MATRIX?

Consider the case of 4 Objectives:

=

4

4

3

4

2

4

1

4

4

3

3

3

2

3

1

3

4

2

3

2

2

2

1

2

4

1

3

1

2

1

1

1

ww

ww

ww

ww

ww

ww

ww

ww

ww

ww

ww

ww

ww

ww

ww

ww

A



STEP 3. Calculate:

jj

N

iiN

i j

iN

iji ww

w

wwa 11

11, ===

∑∑∑ =

==

STEP 4. Calculate Normalized matrix ANorm with elements

i

j

j

i

j

jiN

iji

jiji w

w

ww

w

a

a

aa ====

∑=

11~ ,

1,

,,



=

4444

3333

2222

1111

wwwwwwwwwwwwwwww

ANorm

If Subjective Judgment is perfect the top

the elements in each row should be the same



ANSWER MATRIX

Salary Life quality Work interest Near family Salary 1 5 2 4 Life quality 1/5 1 1/2 1/2 Work interest 1/2 2 1 2 Near family 1/4 2 1/2 1 Sum 1.95 10 4 7.5

“NORMALIZED MATRIX”

Salary Life quality Work interest Near family

Salary 0.513 0.500 0.500 0.533

Life quality 0.103 0.100 0.125 0.067

Work interest 0.256 0.200 0.250 0.267

Near family 0.128 0.200 0.125 0.133

Sum 1.0000 1.0000 1.0000 1.0000



CONCLUSION: Subjective Judgment is NOT PERFECT

STEP 4. Estimate Weight iw for objective i such that:

∑=

=N

jjii a

Nw

1,

~1

Note that:

∑ ∑∑= ==

=

N

i

N

jji

N

ii a

Nw

1 1,

1

~1



∑∑∑

∑∑∑ = =

=

= =

=

=

=N

j

N

iN

iji

jiN

i

N

jN

iji

ji

a

aNa

aN 1 1

1,

,

1 1

1,

, 11

∑∑ ∑∑ == =

=

==

=

N

j

N

j

N

ijiN

iji

Na

aN 11 1,

1,

11111



Back to our Job Selection Example:

Normalized matrix

Salary Life quality Work interest Near family Weights

Salary 0.513 0.500 0.500 0.533 0.5115

Life quality 0.103 0.100 0.125 0.067 0.0986

Work interest 0.256 0.200 0.250 0.267 0.2433

Near family 0.128 0.200 0.125 0.133 0.1466

Sum 1.0000 1.0000 1.0000 1.0000 1



CHECKING CONSISTENCY IN JUDGMENTS

HENCE!

=

4

4

3

4

2

4

1

4

4

3

3

3

2

3

1

3

4

2

3

2

2

2

1

2

4

1

3

1

2

1

1

1

ww

ww

ww

ww

ww

ww

ww

ww

ww

ww

ww

ww

ww

ww

ww

ww

A 1 2 3 4( , , , )Tw w w w w=

1

2

3

4

4444

ww

Awww

=



THEREFORE:

4

4

4

4

4

4

4

3

3

2

2

1

1

=

=

wwwwwwww

wwA




Product Ratios

2.0774 4.0611

0.3958 4.0161

0.9894 4.0672

0.5933 4.0459

CI 0.0159

Consistency Index:

1-nn - s Ratio'Average

=CI

Note That: When expert judgment is perfect CI should be zero



• Suppose an Expert would be filling out the questionnaire at random and we

would calculate the associated CI, what would that value be?

The experiment above can be conducted using computerized random answer. By

conducting this experiment a great number of times one can calculate:

RI – INDEX = average CI – index.

If pairwise comparison matrix is an N*N Matrix the following RI’ indices have

been calculated:

n 2 3 4 5 6 7 8 9 10 RI 0 0.58 0.9 1.12 1.24 1.32 1.41 1.45 1.51



THOMAS SAATY’s suggests:

>

<

nciesInconsiste Serious10.0

rysatisfactoy consistenc of Degree10.0

RICIRICI


• CI = 0.0159

• RI=0.9

• CI/RI=0.0176

CONCLUSION : JUDGEMENT IS CONSISTENT



PAIRWISE COMPARISON MATRIX OF ALTERNATIVES TO OBTAIN SCORES FOR ALTERNATIVE IN EACH OBJECTIVE

Suppose we have N objectives and M alternatives. Then consider the N matrices

Ai of size M*M, such that:

=

MMM

M

i

aa

aaA

,1,

,11,1

where

ai,j= “How much more important is alternative

i compared to to alternative j in objective i.



<

>=

j eAlternativ asImportant less is i eAlternativ1j eAlternativ asImportant as is i eAtlernativ1

j eAlterantiv thanimportant more is i eAlternativ1

, jia

ASSUMPTION:

• There exists a set of numbers iw , i=1,…, n such that:

iw = Relative importance of alternative i compared to the other objectives.

j

ijii

n

ii w

waww =>=∑=

,1

,0,1



Note that:

ji

j

ii

jij a

www

wa

,,

11===

BUT HOW DO WE GET SPECIFIC VALUES FOR ai,j?

ANSWER:

Same as before through

Paired Comparison Questionnaire



OBJECTIVE: STARTING SALARYObjective ObjectiveJob 1 Job 2

LHS is Preferred RHS is Preferred

9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9

Abs

olut

ely

Mor

e Im

porta

nt

Ver

y S

trong

ly M

ore

Impo

rtan

Stro

ngly

Mor

e Im

porta

nt

Slig

htly

Mor

e Im

porta

nt

Equ

ally

Impo

rtant

Slig

htly

Mor

e Im

porta

nt

Stro

ngly

Mor

e Im

porta

nt

Ver

y S

trong

ly M

ore

Impo

rtan

Abs

olut

ely

Mor

e Im

porta

nt

using analytical hierarchy process to select a job

Documents